Message duplicate removal method and device, electronic equipment and storage medium

Abstract

One or more embodiments of the invention provide a message deduplication method and device, electronic equipment and a storage medium. The method comprises the following steps: inputting a to-be-deduplicated message into a text embedding model to obtain an embedding vector; the text embedding model comprises a BGE-M3 sub-model, and the embedding vector fuses semantic features of the message to be subjected to duplicate removal; inputting the embedded vector into a hash value generation model to obtain a hash value of the to-be-deduplicated message; performing screening according to the hash value to obtain an intermediate message subjected to rough message duplicate removal; and performing semantic similarity-based fine message de-duplication on the intermediate message to obtain a final message. According to the invention, the message deduplication accuracy and efficiency can be effectively improved.

Description

Technical Field

[0001] One or more embodiments of the present disclosure relate to the field of data processing technology, and in particular, to a message deduplication method, device, electronic device, and storage medium. Background Art

[0002] In the daily operations of instant messaging platforms, redundant data issues have a significant impact on system performance and user experience from multiple dimensions.

[0003] First, regarding user experience, when users search for specific information, the system must locate valid data amidst a vast amount of repetitive content, significantly increasing search time. For example, a user searching for an important notification may have to sift through dozens of similar "Received" replies, significantly reducing communication efficiency. Secondly, it consumes computing resources. Redundant data not only prolongs the training cycle for models like recommendation systems and natural language processing, but also forces development teams to annotate several times more sample data than necessary to ensure model accuracy. This duplication of effort increases both server computing power costs and the labor cost of manual annotation. Finally, regarding statistical analysis, frequently repeated invalid information can distort the true distribution of keywords. For example, a brand marketing campaign might cause the word "discount" to be mentioned repeatedly within a short period of time, distorting the user interest graph based on word frequency analysis and, in turn, affecting the platform's accurate assessment of user needs. These issues collectively constitute the core challenges of data governance for current instant messaging platforms.

[0004] The high-frequency repetitive short text messages that appear on instant messaging platforms (such as group chat screen-sweeping and emoticon bombardment) have become the main manifestation of data redundancy. Summary of the Invention

[0005] In view of this, an object of one or more embodiments of the present disclosure is to provide a message deduplication method, apparatus, electronic device, and storage medium to solve the problems raised in the background technology.

[0006] Based on the above objectives, one or more embodiments of the present disclosure provide a message deduplication method, including:

[0007] Inputting the message to be deduplicated into a text embedding model to obtain an embedding vector; the text embedding model includes a BGE-M3 sub-model, and the embedding vector integrates the semantic features of the message to be deduplicated;

[0008] Inputting the embedding vector into a hash value generation model to obtain a hash value of the message to be deduplicated;

[0009] Filtering is performed according to the hash value to obtain an intermediate message that has undergone coarse deduplication of messages;

[0010] The intermediate messages are subjected to message deduplication based on semantic similarity to obtain a final message.

[0011] Optionally, the text embedding module further includes a word frequency inverse document frequency feature generation submodule and a splicing layer;

[0012] The step of inputting the message to be deduplicated into the text embedding model to obtain an embedding vector includes:

[0013] Inputting the to-be-deduplicated message into the BGE-M3 sub-model to obtain a learned embedding vector of the to-be-deduplicated message, wherein the learned embedding vector reflects the semantic similarity of the to-be-deduplicated message;

[0014] Inputting the message to be deduplicated into the word frequency inverse document frequency feature generation submodule to obtain the word frequency inverse document frequency feature of the message to be deduplicated;

[0015] The learned embedding vector and the term frequency inverse document frequency feature are input into the concatenation layer to obtain the embedding vector.

[0016] Optionally, before inputting the message to be deduplicated into the text embedding model, the method further includes:

[0017] Classifying the messages to be deduplicated to obtain labels of the messages to be deduplicated;

[0018] A corresponding target text embedding model is selected according to the label of the message to be deduplicated, and parameters of the target text embedding model are adjusted according to the label.

[0019] Optionally, before inputting the message to be deduplicated into the text embedding model, the method further includes:

[0020] The message to be deduplicated is input into a text input data preprocessing module to obtain preprocessed message to be deduplicated, wherein the preprocessing operation includes at least one of data cleaning, word segmentation filtering and stop word filtering.

[0021] Optionally, the hash value generation model is a multilayer perceptron including a first fully connected layer and a second fully connected layer;

[0022] Inputting the embedding vector into a hash value generation model to obtain a hash value of the message to be deduplicated includes:

[0023] Inputting the embedding vector into the first fully connected layer and the second fully connected layer to obtain an intermediate vector; setting the activation function of the first fully connected layer to a rectified linear unit, and setting the activation function of the second fully connected layer to a hyperbolic tangent function or a logistic function;

[0024] The intermediate vector is input into the quantization layer to obtain the hash value.

[0025] Optionally, the loss function of the hash value generation model is:

[0026]

[0027] and,

[0028]

[0029] Among them, Lmain represents the loss value of the main task, L quant Represents the quantization loss value, f(x i ) and f(x j ) represents the text x i and x j A continuous vector, S ij Represents text x i and x j Similarity label, S ij =1 means text x i and x j Similar, S ij =0 text x i and x j Dissimilar, m represents the semantic distance, and the semantic distance between dissimilar texts must be at least m.

[0030] Optionally, it also includes:

[0031] Get the preset whitelist;

[0032] In response to confirming that any message to be deduplicated includes text in the whitelist, the message to be deduplicated is retained.

[0033] Based on the same inventive concept, one or more embodiments of the present disclosure further provide a message deduplication device, including:

[0034] A first computing module is configured to input the message to be deduplicated into a text embedding model to obtain an embedding vector; the text embedding model includes a BGE-M3 sub-model, and the embedding vector incorporates the semantic features of the message to be deduplicated;

[0035] A second computing module is configured to input the embedding vector into a hash value generation model to obtain a hash value of the message to be deduplicated;

[0036] a coarse deduplication module configured to filter according to the hash value to obtain an intermediate message that has undergone coarse deduplication;

[0037] The fine deduplication module is configured to perform fine deduplication on the intermediate messages based on semantic similarity to obtain a final message.

[0038] Based on the same inventive concept, one or more embodiments of the present disclosure also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, the message deduplication method as described in any one of the above items is implemented.

[0039] Based on the same inventive concept, one or more embodiments of the present disclosure further provide a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute any of the above-mentioned message deduplication methods.

[0040] As can be seen from the foregoing, the message deduplication method provided by one or more embodiments of the present disclosure adopts a two-stage architecture of "coarse hash deduplication and fine semantic deduplication." First, a text embedding model based on the BGE-M3 model is used to convert the message into a high-dimensional vector that incorporates semantic features. This text embedding model can capture contextual semantic information. Subsequently, the vector is converted into a hash code using a hash value generation model, and coarse hash deduplication is implemented to form an intermediate candidate set. Finally, fine semantic deduplication is performed to further improve the deduplication effect.

[0041] The disclosed technical solution deeply integrates the core advantages of the BGE-M3 model in the coarse deduplication hashing stage, generating a hash code that incorporates semantic features. This ensures efficient complexity matching while achieving precise filtering in cross-language contexts. This hash generation mechanism, based on deep semantic understanding, enables the coarse deduplication hashing stage to not only quickly filter out completely duplicate messages but also identify semantically similar content expressed in different ways, providing a cleaner candidate set for subsequent fine deduplication. Ultimately, it achieves dual optimization of efficiency and precision in complex cross-language scenarios.

[0042] The message deduplication device, electronic device and computer-readable storage medium provided by the present disclosure can all implement the steps of the above-mentioned message deduplication method, and therefore also have the beneficial effects of the above-mentioned message deduplication method. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate one or more embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only one or more embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 A flowchart of a message deduplication method according to one or more embodiments of the present disclosure is provided;

[0045] Figure 2A schematic structural diagram of a message deduplication device according to one or more embodiments of the present disclosure;

[0046] Figure 3 A schematic diagram of the hardware structure of an electronic device according to one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0047] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0048] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of the present disclosure should have the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in one or more embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0049] As described in the background technology section, the redundant data problem of instant messaging platforms has a significant impact on system performance and user experience from multiple dimensions.

[0050] Currently, text similarity technology is mainly divided into two systems. One is based on the Vector Space Model (VSM) method to calculate the matching degree of texts through cosine similarity.

[0051] For example, hashing technology can be used to map text into compact binary hash codes, enabling fast retrieval. However, this approach has the following technical drawbacks: traditional hashing technology cannot effectively handle polysemy, synonyms, and contextual dependencies; it lacks optimization for short text features, making it difficult to balance retrieval efficiency and semantic preservation.

[0052] Another semantic-based method calculates the similarity of texts through semantic similarity.

[0053] For example, relying on technologies such as keyword matching and TF-IDF, the semantic similarity between texts is reflected by calculating the frequency and weight of keywords in the text. Alternatively, word embedding models such as Word2Vec and GloVe are used to map vocabulary to a high-dimensional vector space to capture richer semantic information. Furthermore, based on pre-trained language models such as BERT and GPT, word vectors are dynamically generated through contextual information, which can effectively handle complex semantic relationships and language transformations. This type of method can better understand and process the implicit semantics in the text and is particularly suitable for application in scenarios with polysemous words, synonyms, and rich contextual information. However, this method still has problems such as slow computing speed and high storage cost.

[0054] In the process of realizing this application, the applicant discovered that the messages on the instant messaging platform have significant multimodal, fragmented and real-time characteristics. These characteristics cause the message data to present the typical characteristics of high noise, high redundancy and low semantic density. As mentioned above, the existing VSM-based hashing method cannot handle the polysemy and synonyms that are prevalent in instant messaging, and lacks feature optimization for ultra-short messages. The semantic method cannot meet the real-time requirements of instant messaging when processing millions of messages. The deep semantic encoding of the BGE-M3 model can resolve the ambiguity of polysemy, hash quantization optimization can achieve real-time processing, and multimodal fusion can improve the matching accuracy of complex messages, providing a breakthrough solution for the redundant data governance of instant messaging platforms.

[0055] Therefore, the applicant provides a message deduplication solution. In this solution, a text embedding model based on the BGE-M3 model is first used to generate a high-dimensional vector that integrates contextual semantics to handle polysemous word ambiguity and long text features. Subsequently, a hash code is generated to achieve fast complexity matching and filter out completely duplicate or highly similar messages. Finally, semantic similarity is calculated for the intermediate candidate set. Through the solution disclosed in this disclosure, the deep semantic understanding capability of the BGE-M3 model can be used to address the semantic preservation defects of traditional hashing technology. By combining coarse hash deduplication and fine semantic deduplication, the efficiency and accuracy of message deduplication can be improved.

[0056] refer to Figure 1 The message deduplication method of one or more embodiments of the present disclosure includes the following steps:

[0057] Step S101: Input the message to be deduplicated into a text embedding model to obtain an embedding vector; the text embedding model includes a BGE-M3 sub-model, and the embedding vector integrates the semantic features of the message to be deduplicated;

[0058] Step S102: inputting the above embedding vector into a hash value generation model to obtain the hash value of the above message to be deduplicated;

[0059] Step S103: Filter according to the above hash value to obtain the intermediate message after the message is roughly deduplicated;

[0060] Step S104: Deduplication of the intermediate messages based on semantic similarity is performed to obtain a final message.

[0061] In the implementation of this disclosure, to reduce computational complexity and improve computational efficiency, a coarse deduplication process is performed using hashing technology before fine semantic deduplication. However, to further improve the accuracy of this coarse deduplication process, this disclosure utilizes a text embedding model based on the BGE-M3 model to obtain an embedding vector, and then generates a hash value based on the embedding vector.

[0062] BGE-M3 is a highly efficient semantic embedding model that extracts deep semantic information and contextual relationships from text, generating precise "learning-based" text embeddings. Building on traditional semantic embeddings, this model "learns" the importance of relevant tokens, even those that have never appeared before, through a deep learning model. This text embedding combines the advantages of semantic embedding and keyword-based embedding, offering strong expressiveness and semantic relevance, enabling efficient retrieval on large amounts of text.

[0063] BGE-M3 integrates corpora of multiple languages ​​during pre-training, and can also capture the commonalities and differences between different languages, thereby generating cross-lingual embedding vectors.

[0064] In the implementation of the present disclosure, after obtaining a learned embedding vector through the BGE-M3 model, a hash value may be generated according to the learned embedding vector.

[0065] In the implementation of the present disclosure, the text embedding model may also include a term frequency inverse document frequency feature generation submodule and a splicing layer. The term frequency inverse document frequency feature generation submodule generates a term frequency inverse document frequency feature (TF-IDF) based on the messages to be deduplicated. Given the characteristics of instant messaging platforms, such as short text, high frequency, and fragmented content, TF-IDF can accurately solve the problem of high-frequency duplication of short text on instant messaging platforms through lightweight feature extraction and efficient similarity calculation, becoming an important tool for message management, content filtering, and retrieval.

[0066] In the embodiments of the present disclosure, the TF-IDF method can be used to limit the vocabulary to 50,000 high-frequency words to generate sparse vectors, and then dimensionality reduction can be used to obtain 256-dimensional statistical features. For long texts such as group chat logs, a global representation can be obtained by truncation or segmentation followed by averaging, providing a reliable semantic foundation for subsequent analysis.

[0067] In this way, the message to be deduplicated is input into the text embedding model to obtain the embedding vector, which can also include: inputting the above-mentioned message to be deduplicated into the above-mentioned BGE-M3 sub-model to obtain the learned embedding vector of the above-mentioned message to be deduplicated; inputting the above-mentioned message to be deduplicated into the above-mentioned word frequency inverse document frequency feature generation sub-module to obtain the word frequency inverse document frequency feature of the above-mentioned message to be deduplicated; inputting the above-mentioned learned embedding vector and the above-mentioned word frequency inverse document frequency feature into the above-mentioned splicing layer to obtain the above-mentioned embedding vector.

[0068] The embedding vectors obtained through the above steps complement the strengths of semantic information and statistical features. On the one hand, BGE-M3's semantic understanding capabilities complement TF-IDF's shortcomings in processing semantics, enabling accurate identification of duplicate messages even when the textual expressions differ but the semantics are similar. On the other hand, TF-IDF's key feature extraction capabilities provide additional information to BGE-M3's embedding vectors, enhancing their discriminability.

[0069] In an embodiment of the present disclosure, the above-mentioned learning embedding model can be 1024-dimensional, and the TF-IDF feature can be 256-dimensional. After splicing, a 1280-dimensional embedding vector can be obtained.

[0070] In the implementation of the present disclosure, the hash value generation model can be a multi-layer perceptron consisting of a first fully connected layer, a second fully connected layer, and a quantization layer. The first fully connected layer can have an embedding vector as input and a first eigenvector as output; the second fully connected layer can have a first eigenvector as input and a second eigenvector as output; and the quantization layer can have a second eigenvector as input and a discrete hash code as output. The purpose of providing two fully connected layers is to gradually reduce the dimensionality of high-dimensional text embeddings and map them to a discrete space, facilitating the subsequent calculation of the Hamming distance.

[0071] The hash value generation module in the disclosed implementation adopts a multi-layer perceptron (MLP) architecture containing two fully connected layers mainly for the following technical considerations: First, the two-layer structure enhances the feature expression capability through staged nonlinear mapping. The first layer reduces the 1024-dimensional semantic vector generated by the text embedding model to 512 dimensions, forming a feature bottleneck while compressing redundant information, forcing the model to learn a more compact semantic representation. The second layer is then mapped to the target hash code dimension (such as 64 bits). This progressive dimensionality reduction strategy combined with the ReLU activation function can more flexibly fit the complex nonlinear relationship of high-dimensional continuous embedding to low-dimensional binary code. According to the universal approximation theorem, the two-layer network is easier to achieve stable training than the single-layer structure while ensuring the model capacity. Its gradient propagation path is shorter and the parameter update is smoother, which effectively alleviates the gradient vanishing problem caused by direct large-span mapping and makes the hash code generation process easier to converge.

[0072] In an embodiment of the present disclosure, the activation function of the first fully connected layer can be ReLU, the activation function of the second fully connected layer can be tanh or sigmoid, and the elements are restricted to the interval [-1,1]. The quantization layer can quantize the second eigenvector into a binary code (that is, mapped into a discrete hash code) through the sign function.

[0073] In the implementation of this disclosure, the loss function of the hash value generation model can be divided into two parts: the main task loss and the quantization loss. The main task loss is used to ensure that similar texts are close to each other in the hash space, while the quantization loss is used to constrain the MLP output to be as close to the ideal binary state as possible. Through the appropriate design of the loss function, semantic similarity and quantization error are balanced, ensuring consistency and discriminability between continuous embeddings and discrete hash codes during end-to-end training.

[0074] In the implementation of the present disclosure, the loss function of the hash value generation model can be set as:

[0075]

[0076] and,

[0077]

[0078] Among them, Lmain represents the loss value of the main task, L quant Represents the quantization loss value, f(x i ) and f(x j ) represents the text x i and x j A continuous vector, S ij Represents text x i and x j Similarity label, S ij =1 means text x i and x j Similar, S ij =0 text x i and x j Dissimilar, m represents the semantic distance, and the semantic distance between dissimilar texts must be at least m.

[0079] The first loss function is designed to preserve semantic similarity. Choosing either contrastive loss or triplet loss ensures that semantically similar texts have a minimal Hamming distance in the hash space. Due to the inherent semantic sparsity and diversity of text data, this loss is better able to capture semantic information.

[0080] The setting of the second loss function is to quantify the error constraint. Since the sign function is non-differentiable, the tanh approximation is usually adopted, and a quantization regularization term (such as binary cross-entropy loss) is introduced to ensure that the continuous MLP output approaches the ideal binary value, guaranteeing smooth gradients and end-to-end optimization effects.

[0081] In the implementation of the present disclosure, in order to further improve the deduplication effect, data preprocessing can also be performed on the message to be deduplicated before it is input into the text embedding model.

[0082] In the embodiments of the present disclosure, data preprocessing may include stop word processing, rule filtering, etc.

[0083] Specifically, stop word processing is to clean the noise in the message text, including common stop words (such as "of", "is", "in", etc.), web page elements (such as HTML tags, JavaScript code, etc.), and meaningless symbols (such as emojis, special characters, etc.). Stop word processing helps to reduce the data volume in subsequent processing steps and improve the accuracy of analysis. In the embodiments of the present disclosure, a pre-defined stop word list and regular expressions can be used to identify and remove these elements.

[0084] In the embodiments of the present disclosure, Chinese word segmentation tools such as jieba can be selected to accurately segment the text, and a custom dictionary can be introduced to improve the recognition accuracy of professional terms and new words.

[0085] Through stop word processing, noise can be effectively reduced, providing more valuable information for subsequent processing.

[0086] Rule filtering aims to identify and exclude messages to be deduplicated that follow specific generation patterns, such as advertisement information. Such messages to be deduplicated usually try to bypass the conventional filtering mechanism by changing the sentence structure, adding specific characters, etc. Therefore, rule filtering can be achieved by setting a series of matching patterns or rule sets. When the message to be deduplicated conforms to a certain or certain rules, the message to be deduplicated is marked as spam and excluded.

[0087] The present disclosure does not limit the specific data preprocessing method, and different data preprocessing methods are within the protection scope of the present disclosure.

[0088] In the implementation of the present disclosure, the message to be deduplicated can also be classified before it is input into the text embedding model. This can help to more accurately understand the message background, thereby effectively performing the deduplication operation. For example, the word "apple" may have different meanings (food or electronic technology brand) in different contexts. By classifying the message to be deduplicated, the text meaning can be better identified according to the scenario.

[0089] In the embodiments of the present disclosure, messages can be categorized based on multiple dimensions, such as time, user, group, and topic. For example, messages with the same topic but posted at different times can be grouped based on posting time; messages from different groups with similar content can be differentiated based on group attributes.

[0090] This disclosure does not limit the classification method, and different classification methods are all within the protection scope of this disclosure.

[0091] Furthermore, to prevent accidental deletion of important messages, embodiments of the present disclosure can also establish a keyword library and whitelist for keywords or key text. The keyword library can help the system identify which content is important and should not be accidentally deleted. Based on these keywords, whitelist rules can be further constructed to ensure that relevant important information is not mistakenly marked as duplicate or spam during the deduplication process.

[0092] It can be understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities.

[0093] It should be noted that the methods of one or more embodiments of the present disclosure can be performed by a single device, such as a computer or server. The methods of these embodiments can also be applied in a distributed scenario, performed by multiple devices working together. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the methods of one or more embodiments of the present disclosure, and the multiple devices will interact with each other to complete the described methods.

[0094] It should be noted that the above description is of specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0095] Based on the same inventive concept, corresponding to any of the above embodiments and methods, the present disclosure also provides a message deduplication device. Figure 2 As shown, the device includes:

[0096] The first computing module 11 is configured to input the message to be deduplicated into a text embedding model to obtain an embedding vector; the text embedding model includes a BGE-M3 sub-model, and the embedding vector integrates the semantic features of the message to be deduplicated;

[0097] A second calculation module 12 is configured to input the embedding vector into a hash value generation model to obtain a hash value of the message to be deduplicated;

[0098] A coarse deduplication module 13 is configured to filter according to the hash value to obtain an intermediate message that has undergone coarse deduplication;

[0099] The fine deduplication module 14 is configured to perform fine deduplication on the intermediate messages based on semantic similarity to obtain a final message.

[0100] For the convenience of description, the above devices are described as being functionally divided into various modules. Of course, when implementing one or more embodiments of the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0101] The apparatus of the above embodiment is used to implement the corresponding method in the above embodiment and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0102] Figure 3 10 is a schematic diagram showing a more specific hardware structure of an electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are communicatively connected to each other within the device via the bus 1050.

[0103] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present disclosure.

[0104] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present disclosure are implemented through software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0105] The input / output interface 1030 is used to connect an input / output module to implement information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0106] The communication interface 1040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0107] The bus 1050 comprises a path for transmitting information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0108] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, those skilled in the art will understand that the above device may only include the components necessary to implement the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0109] The electronic devices of the above embodiments are used to implement the corresponding methods in the above embodiments and have the beneficial effects of the corresponding method embodiments, which will not be described in detail here.

[0110] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0111] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Within the scope of the present disclosure, the technical features of the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0112] In addition, to simplify the description and discussion, and so as not to obscure one or more embodiments of the present disclosure, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, devices may be shown in the form of block diagrams to avoid obscuring one or more embodiments of the present disclosure, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which one or more embodiments of the present disclosure will be implemented (i.e., these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that one or more embodiments of the present disclosure may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0113] Although the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0114] The one or more embodiments of the present disclosure are intended to encompass all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the one or more embodiments of the present disclosure should be included within the scope of protection of the present disclosure.

Claims

1. A message deduplication method, characterized in that: include: Input the message to be deduplicated into the text embedding model to obtain the embedding vector; The text embedding model includes a BGE-M3 sub-model, and the embedding vector integrates the semantic features of the message to be deduplicated; Inputting the embedding vector into a hash value generation model to obtain a hash value of the message to be deduplicated; Filtering is performed according to the hash value to obtain an intermediate message that has undergone coarse deduplication of messages; The intermediate messages are subjected to message deduplication based on semantic similarity to obtain a final message.

2. The method according to claim 1, characterized in that The text embedding module also includes a word frequency inverse document frequency feature generation submodule and a splicing layer; The step of inputting the message to be deduplicated into the text embedding model to obtain an embedding vector includes: Inputting the to-be-deduplicated message into the BGE-M3 sub-model to obtain a learned embedding vector of the to-be-deduplicated message, wherein the learned embedding vector reflects the semantic similarity of the to-be-deduplicated message; Inputting the message to be deduplicated into the word frequency inverse document frequency feature generation submodule to obtain the word frequency inverse document frequency feature of the message to be deduplicated; The learned embedding vector and the term frequency inverse document frequency feature are input into the concatenation layer to obtain the embedding vector.

3. The method according to claim 1 or 2, characterized in that Before inputting the message to be deduplicated into the text embedding model, it also includes: Classifying the messages to be deduplicated to obtain labels of the messages to be deduplicated; A corresponding target text embedding model is selected according to the label of the message to be deduplicated, and parameters of the target text embedding model are adjusted according to the label.

4. The method according to claim 3, characterized in that Before inputting the message to be deduplicated into the text embedding model, it also includes: The message to be deduplicated is input into a text input data preprocessing module to obtain preprocessed message to be deduplicated, wherein the preprocessing operation includes at least one of data cleaning, word segmentation filtering and stop word filtering.

5. The method according to claim 1, wherein The hash value generation model is a multi-layer perceptron including a first fully connected layer and a second fully connected layer; Inputting the embedding vector into a hash value generation model to obtain a hash value of the message to be deduplicated includes: Inputting the embedding vector into the first fully connected layer and the second fully connected layer to obtain an intermediate vector; setting the activation function of the first fully connected layer to a rectified linear unit, and setting the activation function of the second fully connected layer to a hyperbolic tangent function or a logistic function; The intermediate vector is input into the quantization layer to obtain the hash value.

6. The method according to claim 5, characterized in that The loss function of the hash value generation model is: and, Among them, Lmain represents the main task loss value, L quant Represents the quantization loss value, f(x i ) and f(x j ) represents the text x i and x j A continuous vector, S ij Represents text x i and x j Similarity label, S ij =1 means text x i and x j Similar, S ij =0 text x i and x j Dissimilar, m represents the semantic distance, and the semantic distance between dissimilar texts must be at least m.

7. The method according to claim 1, characterized in that Also includes: Get the preset whitelist; In response to confirming that any message to be deduplicated includes text in the whitelist, the message to be deduplicated is retained.

8. A message deduplication device, characterized in that: include: A first computing module is configured to input the message to be deduplicated into a text embedding model to obtain an embedding vector; The text embedding model includes a BGE-M3 sub-model, and the embedding vector integrates the semantic features of the message to be deduplicated; A second computing module is configured to input the embedding vector into a hash value generation model to obtain a hash value of the message to be deduplicated; a coarse deduplication module configured to filter according to the hash value to obtain an intermediate message that has undergone coarse deduplication; The fine deduplication module is configured to perform fine deduplication on the intermediate messages based on semantic similarity to obtain a final message.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

Images