Text deduplication method and device, equipment, storage medium and program product
By comprehensively utilizing a multi-dimensional deduplication method that combines hash mapping tables, cosine similarity, and visual feature matching, the problem of duplicate resource uploads in rich text editors is solved, achieving more efficient storage space utilization and content management.
Patent Information
- Application Number
- CN202511258731.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-04
- Publication Date
- 2025-12-16
AI Technical Summary
Existing deduplication methods mainly rely on hash comparison technology, which cannot effectively handle the problem of duplicate uploads of the same or highly similar resources, resulting in wasted server storage space and a chaotic content resource library.
By combining hash mapping tables, cosine similarity calculation, and visual feature matching, the system comprehensively judges the overlapping information of the text to be processed, including hash value comparison, semantic similarity, and visual feature matching, and performs multi-dimensional deduplication.
It improves the accuracy and comprehensiveness of deduplication, avoids the waste of storage space and content management chaos caused by repeated uploads, and improves the efficiency and quality of content processing.
Smart Images
Figure CN121145858A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of artificial intelligence, fintech, or audio-visual technology, and in particular to a text deduplication method, apparatus, device, storage medium, and program product. Background Technology
[0002] Rich text editors, as tools that support diverse content editing, allow users to flexibly add elements such as images, videos, audio, tables, and specially formatted text while editing text.
[0003] When using rich text editors, users frequently upload images, videos, and other content resources, which can easily lead to duplicate uploads of the same or highly similar resources. This not only consumes a large amount of server storage space but also causes duplicate files to accumulate in the backend content resource library.
[0004] Existing deduplication methods primarily rely on hash comparison technology, which calculates the hash value of files to determine if they are identical. However, this method is relatively simplistic and cannot handle all situations, leading to the continued problem of duplicate uploads. Summary of the Invention
[0005] This application provides a text deduplication method, apparatus, device, storage medium, and program product to solve the problem of duplicate uploads.
[0006] Firstly, this application provides a text deduplication method, including:
[0007] Get the text to be processed;
[0008] Based on a pre-set hash mapping table and the hash value corresponding to at least one pre-stored reference text, the first overlapping information corresponding to the text to be processed is determined;
[0009] Based on the cosine similarity between the text to be processed and at least one pre-stored reference text, the second overlap information corresponding to the text to be processed is determined;
[0010] Match at least one pre-stored candidate resource feature with the visual features corresponding to the text to be processed to determine the third overlapping information corresponding to the text to be processed;
[0011] The text to be processed is deduplicated by combining at least one of the first overlapping information, the second overlapping information, and the third overlapping information.
[0012] In one possible implementation, determining the first overlap information corresponding to the text to be processed based on a pre-set hash mapping table and the hash value corresponding to at least one pre-stored reference text includes:
[0013] Extract at least one text tag contained in the text to be processed, and concatenate them to form the string to be processed;
[0014] The string to be processed is processed using a preset hash function to obtain the target hash value corresponding to the text to be processed;
[0015] The prefix of the target hash value is matched with the hash value corresponding to the reference text, and the matching result is used as the first overlap information.
[0016] In one possible implementation, determining the second overlap information corresponding to the text to be processed based on the cosine similarity between the text to be processed and at least one pre-stored reference text includes:
[0017] The text to be processed is input into an image recognition model to obtain multiple candidate labels output by the image recognition model, and the label confidence score corresponding to each candidate label;
[0018] For any candidate label, if the label confidence of the candidate label meets the preset confidence level, the candidate label is used as the target label;
[0019] The second overlap information corresponding to the text to be processed is determined based on the cosine similarity between the target label and at least one pre-stored reference text.
[0020] In one possible implementation, determining the cosine similarity between the target tag and at least one pre-stored reference text includes:
[0021] The target label is input into the first word vector model to obtain the first word vector of the target label;
[0022] Input at least one pre-stored reference text into the second word vector model to obtain at least one second word vector;
[0023] At least one cosine similarity is determined based on the first word vector and at least one of the second word vectors.
[0024] In one possible implementation, matching at least one pre-stored candidate resource feature with the visual features corresponding to the text to be processed to determine the third overlap information corresponding to the text to be processed includes:
[0025] Match at least one pre-stored candidate resource feature with the visual features of the text to be processed to obtain at least one feature matching value;
[0026] The third overlap information is determined when at least one of the feature matching values satisfies a preset feature matching value.
[0027] In one possible implementation, matching at least one pre-stored candidate resource feature with the visual features of the text to be processed to obtain at least one feature matching value includes:
[0028] At least one of the candidate resource features is vectorized to obtain at least one candidate resource feature vector;
[0029] The visual features of the text to be processed are vectorized to obtain visual feature vectors;
[0030] The at least one feature matching value is determined based on the at least one candidate resource feature vector and the visual feature vector.
[0031] In one possible implementation, after deduplication of the text to be processed, the method further includes:
[0032] Control the upload of the text to be processed;
[0033] The text to be processed is stored as a new reference text.
[0034] Secondly, this application provides a text deduplication device, comprising:
[0035] The acquisition module is used to acquire the text to be processed;
[0036] The determination module is used to determine the first overlap information corresponding to the text to be processed based on a pre-set hash mapping table and the hash value corresponding to at least one pre-stored reference text.
[0037] The determining module is further configured to determine the second overlap information corresponding to the text to be processed based on the cosine similarity between the text to be processed and at least one pre-stored reference text;
[0038] The determining module is also used to match at least one pre-stored candidate resource feature with the visual features corresponding to the text to be processed, and determine the third overlapping information corresponding to the text to be processed;
[0039] The processing module is used to perform deduplication processing on the text to be processed by combining at least one of the first overlapping information, the second overlapping information, and the third overlapping information.
[0040] In one possible implementation, the device further includes: a splicing module;
[0041] The splicing module is used to extract at least one text tag contained in the text to be processed and splice them to form the string to be processed;
[0042] The processing module is also used to process the string to be processed using a preset hash function to obtain the target hash value corresponding to the text to be processed;
[0043] The device further includes: a matching module;
[0044] The matching module is used to match the prefix of the target hash value with the hash value corresponding to the reference text, and use the matching result as the first overlap information.
[0045] In one possible implementation, the device further includes: an input module;
[0046] The input module is used to input the text to be processed into the image recognition model to obtain multiple candidate labels output by the image recognition model, and the label confidence level corresponding to each candidate label;
[0047] The determining module is further configured to, for any candidate label, take the candidate label as the target label if the label confidence of the candidate label meets the preset confidence level.
[0048] The determining module is specifically used to determine the second overlap information corresponding to the text to be processed based on the cosine similarity between the target label and at least one pre-stored reference text.
[0049] In one possible implementation, the input module is further configured to input the target label into a first word vector model to obtain a first word vector of the target label;
[0050] The input module is also used to input at least one pre-stored reference text into the second word vector model to obtain at least one second word vector;
[0051] The determining module is specifically used to determine at least one cosine similarity based on the first word vector and at least one second word vector.
[0052] In one possible implementation, the matching module is further configured to match at least one pre-stored candidate resource feature with the visual features of the text to be processed to obtain at least one feature matching value.
[0053] The determining module is specifically used to determine the third overlap information when at least one of the feature matching values satisfies a preset feature matching value.
[0054] In one possible implementation, the processing module is further configured to vectorize at least one of the candidate resource features to obtain at least one candidate resource feature vector.
[0055] The processing module is also used to vectorize the visual features of the text to be processed to obtain a visual feature vector.
[0056] The determining module is specifically used to determine the at least one feature matching value based on the at least one candidate resource feature vector and the visual feature vector.
[0057] In one possible implementation, the device further includes: a control module;
[0058] The control module is used to control the uploading of the text to be processed;
[0059] The determining module is also used to store the text to be processed as a new reference text.
[0060] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;
[0061] The memory stores computer-executed instructions;
[0062] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.
[0063] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.
[0064] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.
[0065] The text deduplication method, apparatus, device, storage medium, and program product provided in this application first acquire the text to be processed. Then, based on a pre-set hash mapping table and the hash values corresponding to at least one pre-stored reference text, the first overlap information of the text to be processed is determined. The second overlap information is determined based on the cosine similarity between the text to be processed and at least one pre-stored reference text. Simultaneously, at least one pre-stored candidate resource feature is matched with the visual features corresponding to the text to be processed to determine the third overlap information. Finally, at least one of the first, second, and third overlap information is combined to perform deduplication on the text to be processed. This method comprehensively utilizes multiple methods such as hash value comparison, cosine similarity calculation, and visual feature matching to determine overlap information. Compared with single deduplication methods, it improves the accuracy and comprehensiveness of deduplication, effectively avoids the waste of storage space and content management chaos caused by repeatedly uploading the same or similar content, and improves the efficiency and quality of content processing. Attached Figure Description
[0066] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0067] Figure 1 Flowchart of the text deduplication method provided in this application Figure 1 ;
[0068] Figure 2 Flowchart of the text deduplication method provided in this application Figure 2 ;
[0069] Figure 3 A schematic diagram of the text deduplication device provided in this application;
[0070] Figure 4 A schematic diagram of the text deduplication device provided in this application.
[0071] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0072] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0073] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize or refuse.
[0074] Furthermore, the technical solution involved in this application, which involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.) and the use of artificial intelligence technology for automated decision-making, and makes decisions that have a significant impact on personal rights based on the results of automated decision-making, provides users with corresponding operation entry points for users to choose to agree to or reject the results of automated decision-making; if the user chooses to reject, the process will proceed to the expert decision-making process.
[0075] It should be noted that the text deduplication method, apparatus, device, storage medium, and program product provided in this application can be used in the fields of artificial intelligence, fintech, or audio-visual, and can also be used in any field other than artificial intelligence, fintech, or audio-visual. The application fields of the text deduplication method, apparatus, device, storage medium, and program product in this application are not limited.
[0076] Rich text editors, as text tools that allow users to create and edit text containing various formatted content, are widely used in platforms such as websites and email systems. With rich text editors, users can not only input text, but also insert various multimedia elements such as images, videos, tables, and links, improving text readability and interactivity, and enhancing the user experience.
[0077] When using rich text editors, users frequently upload images, videos, and other content resources, which can easily lead to duplicate uploads of the same or highly similar resources. This not only consumes a large amount of server storage space but also causes duplicate files to accumulate in the backend content resource library.
[0078] To address the aforementioned issues, this application provides a text deduplication method that obtains the text to be processed; determines the first overlap information corresponding to the text to be processed based on a pre-set hash mapping table and the hash values corresponding to at least one pre-stored reference text; determines the second overlap information corresponding to the text to be processed based on the cosine similarity between the text to be processed and at least one pre-stored reference text; matches at least one pre-stored candidate resource feature with the visual features corresponding to the text to be processed to determine the third overlap information corresponding to the text to be processed; and combines at least one of the first, second, and third overlap information to perform deduplication processing on the text to be processed. This method, through multi-dimensional judgment of text-level hash comparison, cosine similarity analysis, and visual feature matching, can not only completely identify duplicate text and resources but also reduce the storage space occupied by duplicate resources.
[0079] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0080] Figure 1 Flowchart of the text deduplication method provided in this application Figure 1 .like Figure 1 As shown, the method includes...
[0081] S101. Obtain the text to be processed.
[0082] The text to be processed can be, for example, an image or a video. This application does not impose any special restrictions on this.
[0083] Understandably, in rich text editors, users often upload new text. Only by first obtaining the text to be processed can it be determined whether the text is duplicated or highly similar to the text already stored, and thus decide whether to allow the text to be uploaded.
[0084] S102. Based on a pre-set hash mapping table and the hash value corresponding to at least one pre-stored reference text, determine the first overlapping information corresponding to the text to be processed.
[0085] The first overlap information is used to indicate whether the hash value corresponding to the text to be processed is consistent with the hash value corresponding to at least one pre-stored reference text. If the first overlap information indicates that the hash value corresponding to the text to be processed is consistent with the hash value corresponding to a pre-stored reference text, it means that the text to be processed is duplicated; otherwise, if the first overlap information indicates that the hash value corresponding to the text to be processed is consistent with the hash value corresponding to a pre-stored reference text, it means that the text to be processed is not duplicated.
[0086] The purpose of this step is to determine whether the text to be processed is duplicated based on the hash value.
[0087] Understandably, since a pre-set hash map table stores the hash values of all reference text in the rich text editor, by comparing the text to be processed with the hash value of at least one reference text in this hash map table, a comparison result can be obtained. Using this comparison result as the first overlap information, it can be determined whether there are duplicates in the text to be processed.
[0088] S103. Based on the cosine similarity between the text to be processed and at least one pre-stored reference text, determine the second overlap information corresponding to the text to be processed.
[0089] The second overlap information is used to indicate whether the cosine similarity between the text to be processed and at least one pre-stored reference text is greater than a preset cosine similarity. If the second overlap information indicates that the cosine similarity between the text to be processed and at least one pre-stored reference text is greater than the preset cosine similarity, it means that the text to be processed exists; otherwise, if the second overlap information indicates that the cosine similarity between the text to be processed and at least one pre-stored reference text is not greater than the preset cosine similarity, it means that the text to be processed does not exist.
[0090] The purpose of this step is to determine whether the text to be processed is duplicated based on cosine similarity.
[0091] Understandably, by calculating the cosine similarity between the text to be processed and at least one pre-stored reference text, and further determining the similarity between the text to be processed and at least one pre-stored reference text, and generating second overlap information accordingly, it is possible to determine whether there is a duplication in the text to be processed.
[0092] S104. Match at least one pre-stored candidate resource feature with the visual features corresponding to the text to be processed to determine the third overlapping information corresponding to the text to be processed.
[0093] The third overlap information is used to indicate whether the feature matching degree between the visual features corresponding to the text to be processed and at least one pre-stored candidate resource feature is greater than a preset feature matching degree. If the third overlap information indicates that the feature matching degree between the visual features corresponding to the text to be processed and at least one pre-stored candidate resource feature is greater than the preset feature matching degree, it means that the text to be processed is duplicated; conversely, if the third overlap information indicates that the feature matching degree between the visual features corresponding to the text to be processed and at least one pre-stored candidate resource feature is not greater than the preset feature matching degree, it means that the text to be processed is not duplicated.
[0094] The purpose of this step is to determine whether the text to be processed is duplicated based on visual features.
[0095] Understandably, candidate resource features can reflect the visual representation of stored text, and the visual features of the text to be processed can also reflect its visual representation. Therefore, by matching at least one pre-stored candidate resource feature with the visual features corresponding to the text to be processed, the matching result of the text to be processed from a visual perspective can be determined, and the matching result can be used as a third overlap information to determine whether there are duplicates in the text to be processed.
[0096] S105. Combine at least one of the first overlapping information, the second overlapping information, and the third overlapping information to perform deduplication processing on the text to be processed.
[0097] This step can be combined in a way that, for example, judges two or three overlapping pieces of information in a pre-set order to determine whether they are duplicates. For example, the pre-set order is the first overlapping piece of information, the second overlapping piece of information, and the third overlapping piece of information. It can also be done by any one of these methods, or by simultaneously judging two or three pieces of information, such as judging the first overlapping piece of information and the second overlapping piece of information, or judging the first overlapping piece of information and the third overlapping piece of information.
[0098] The purpose of this step is to deduplicate the text to be processed based on multiple dimensions.
[0099] Understandably, since hash values, semantic similarity, and visual similarity measure text similarity from different perspectives, hash value comparison can find completely identical text, cosine similarity can identify semantically similar text, and visual feature comparison can identify visually similar text but different content.
[0100] Therefore, by comprehensively considering the first, second, and third overlapping information, it is possible to more comprehensively determine whether the text to be processed is duplicated or similar, ensuring that the deduplication process is more accurate and effective, and the text ultimately retained is unique and without duplication.
[0101] Optionally, assume the pre-set order is the first overlapping information, the second overlapping information, and the third overlapping information. Then, when comparing in this order, if the first overlapping information indicates a duplicate in the text to be processed, then the text to be processed is confirmed to be duplicated. In this case, the second and third overlapping information do not need to be evaluated again. Similarly, if the first and second overlapping information indicate a duplicate in the text to be processed, the third overlapping information does not need to be evaluated again.
[0102] The text deduplication method provided in this embodiment first obtains the text to be processed. Then, based on a pre-set hash mapping table and the hash values corresponding to at least one pre-stored reference text, it determines the first overlap information of the text to be processed. Based on the cosine similarity between the text to be processed and at least one pre-stored reference text, it determines the second overlap information. Simultaneously, it matches at least one pre-stored candidate resource feature with the visual features corresponding to the text to be processed to determine the third overlap information. Finally, it combines at least one of the first, second, and third overlap information to perform deduplication on the text to be processed. This method comprehensively utilizes multiple methods such as hash value comparison, cosine similarity calculation, and visual feature matching to determine overlap information. Compared to a single deduplication method, it improves the accuracy and comprehensiveness of deduplication, effectively avoids the waste of storage space and content management chaos caused by repeatedly uploading the same or similar content, and improves the efficiency and quality of content processing.
[0103] Figure 2 Flowchart of the text deduplication method provided in this application Figure 2 ,like Figure 2 As shown, in this embodiment... Figure 1 Based on the examples, the text deduplication method is described in detail, which includes:
[0104] S201. Obtain the text to be processed.
[0105] The explanation of step S201 is the same as that in the above embodiments, and will not be repeated here.
[0106] S202. Extract at least one text tag contained in the text to be processed and concatenate them to form the string to be processed.
[0107] The purpose of this step is to obtain key text tags from the text to be processed.
[0108] Understandably, since the text to be processed may contain redundant information, and text tags can reflect the core characteristics of the text, more representative processing data can be obtained by extracting at least one text tag and concatenating it with the text content to form a new string.
[0109] S203. Use a preset hash function to process the string to be processed to obtain the target hash value corresponding to the text to be processed.
[0110] The preset hash function is used to map the input string to be processed into a hash value of fixed length.
[0111] The purpose of this step is to generate a unique identifier for the text to be processed.
[0112] Understandably, by using a preset hash function to process the string to be processed, a unique identifier, namely the target hash value, can be generated for the text to be processed. This target hash value can quickly determine whether the same text exists in the text editor in subsequent text comparisons.
[0113] S204. Match the prefix of the target hash value with the hash value corresponding to the reference text, and use the matching result as the first overlapping information.
[0114] The matching result indicates whether the hash value corresponding to the reference text has the same prefix as the target hash value. If the matching result indicates that the hash value corresponding to the reference text has the same prefix as the target hash value, it means that the text to be processed already exists; otherwise, if the matching result indicates that the hash value corresponding to the reference text has a different prefix than the target hash value, it means that the text to be processed does not exist.
[0115] Understandably, since the prefix of the target hash value can represent the text to be processed, by matching the prefix of the target hash value with the hash value corresponding to the reference text, it is possible to quickly determine whether the text to be processed is duplicated based on the matching result, without having to compare the entire text to be processed or the complete hash value, thereby improving matching efficiency.
[0116] S205. Input the text to be processed into the image recognition model to obtain multiple candidate labels output by the image recognition model, as well as the label confidence level corresponding to each candidate label.
[0117] The image recognition model is used to extract key feature information from the text to be processed. The input of the image recognition model is the text to be processed, and the output is multiple candidate labels and the label confidence score corresponding to each candidate label.
[0118] Different candidate labels represent different key text information. For example, a candidate label of "architecture" indicates that the text to be processed is related to architecture, while a candidate label of "outdoors" indicates that the text to be processed is related to outdoor scenes.
[0119] Label confidence is used to characterize the trustworthiness of each candidate label. The higher the label confidence of a candidate label, the higher the trustworthiness of that candidate label; conversely, the lower the label confidence of a candidate label, the lower the trustworthiness of that candidate label.
[0120] Label confidence can be displayed as a percentage, such as 80%, or as a decimal, such as 0.7.
[0121] The purpose of this step is to input the text to be processed into an image recognition model, use the model to analyze the content of the text to be processed, and obtain multiple candidate labels and the confidence level of each candidate label.
[0122] Understandably, image recognition models can identify and extract image-related semantics from the text to be processed. Therefore, by inputting the text to be processed into an image recognition model, the analytical capabilities of the image recognition model can be utilized to determine more critical information from the text.
[0123] For example, suppose we have a video describing natural scenery. Inputting this video into an image recognition model will yield labels related to the terrain, labels related to the scenery, and the confidence score for each label.
[0124] S206. For any candidate label, if the label confidence of the candidate label meets the preset confidence level, the candidate label shall be used as the target label.
[0125] The preset reliability can be displayed as a percentage, such as 90%, or as a decimal, such as 0.8.
[0126] The purpose of this step is to identify candidate labels with sufficiently high confidence from multiple candidate labels.
[0127] Understandably, candidate labels output by image recognition models may not be entirely relevant to the text being processed, and some labels may have low confidence levels, potentially leading to inaccuracies. Therefore, by comparing the confidence level of each candidate label with a pre-set confidence level, labels with higher confidence can be selected and used as the target labels.
[0128] S207. Determine the second overlap information corresponding to the text to be processed based on the cosine similarity between the target label and at least one pre-stored reference text.
[0129] Cosine similarity is used to characterize the degree of similarity between the target label and each pre-stored reference text. A higher cosine similarity between the target label and a pre-stored reference text indicates a higher degree of similarity, suggesting that the text to be processed is duplicated. Conversely, a lower cosine similarity between the target label and a pre-stored reference text indicates a lower degree of similarity, suggesting that the text to be processed is not duplicated.
[0130] The purpose of this step is to determine the degree of similarity between the text to be processed and each pre-stored reference text by using the cosine similarity between the candidate tags and each pre-stored reference text.
[0131] Understandably, since the target labels are obtained through image recognition models, they can represent key information of the text to be processed. Therefore, by calculating the cosine similarity between the target labels with high confidence and each pre-stored reference text, the degree of similarity between the text to be processed and each reference text can be determined, thereby determining whether the text to be processed is duplicated.
[0132] Optionally, this application provides a possible method for determining the cosine similarity between a target tag and at least one pre-stored reference text:
[0133] The first step is to input the target label into the first word vector model to obtain the first word vector of the target label.
[0134] The purpose of this step is to obtain the first word vector of the target label through the first word vector model.
[0135] Understandably, since the target labels are in natural language form and cannot be processed directly, the first word vector model can be used to convert the target labels into digital vector representations, which facilitates subsequent processing and comparison.
[0136] The second step is to input at least one pre-stored reference text into the second word vector model to obtain at least one second word vector.
[0137] The purpose of this step is to obtain the second word vector for each reference text through the second word vector model.
[0138] Understandably, the reference text contains complex semantic information, and its simple text form is insufficient for processing and comparison. Therefore, by using a second word vector model, the semantic information of the reference text can be encoded into continuous numerical vectors, allowing the reference text to be presented in a vectorized form.
[0139] The third step is to determine at least one cosine similarity based on the first word vector and at least one second word vector.
[0140] The purpose of this step is to measure the similarity between the target label and the preset reference text based on the first word vector and the second word vector.
[0141] Understandably, by comprehensively considering the first word vector and the second word vector, the geometric properties of vectors in numerical space can be used to calculate the similarity value between the first word vector and the second word vector, thereby determining the degree of similarity between the target tag and the preset reference text based on the similarity value.
[0142] S208. Match at least one pre-stored candidate resource feature with the visual features of the text to be processed to obtain at least one feature matching value.
[0143] Among them, at least one candidate resource feature stored in advance refers to a set of data that can represent the core visual attributes of the visual resource, extracted in advance from the uploaded and reviewed visual resources (such as images, video frames, GIFs, etc.).
[0144] Feature matching values indicate the degree of match between the visual features of the text to be processed and the features of each candidate resource. Feature matching values can be expressed numerically, such as 0.7, or as a percentage, such as 80%. A higher feature matching value indicates a higher degree of match between the visual features of the text to be processed and the features of each candidate resource, suggesting that the text to be processed is duplicated. Conversely, a lower feature matching value indicates a lower degree of match between the visual features of the text to be processed and the features of each candidate resource, suggesting that the text to be processed is not duplicated.
[0145] The purpose of this step is to identify candidate resources that are visually similar to the text to be processed by matching the features of pre-stored candidate resources with the visual features of the text to be processed.
[0146] Understandably, since the text to be processed may contain implicit visual features, by matching the visual features of the text to be processed with the features of the candidate resources, the degree of similarity between the text to be processed and the candidate resources can be judged more accurately from a visual perspective.
[0147] Optionally, this application provides a possible implementation method, including:
[0148] The first step is to vectorize at least one candidate resource feature to obtain at least one candidate resource feature vector.
[0149] Understandably, candidate resource features may contain various types of information, such as image attributes and audio features, which are difficult to use or understand directly in their raw state. Therefore, vectorization allows candidate resource features to be represented in numerical form.
[0150] The second step is to vectorize the visual features of the text to be processed, thereby obtaining visual feature vectors.
[0151] Understandably, the visual features of the text being processed are difficult to use and understand directly in their original state, such as color and shape features. Therefore, by vectorizing the visual features of the text being processed, these visual features can be represented in numerical form.
[0152] The third step is to determine at least one feature matching value based on at least one candidate resource feature vector and a visual feature vector.
[0153] The purpose of this step is to determine the degree of matching between candidate resource features and visual features.
[0154] Understandably, candidate resource feature vectors reflect the feature information of candidate resources across various dimensions, while visual feature vectors reflect the visual feature information of the text to be processed. Therefore, by determining the feature matching value between candidate resource feature vectors and visual feature vectors, the degree of matching between the candidate resource features and visual features can be determined.
[0155] Optionally, this application provides a method for determining the visual features of the text to be processed, such as inputting the text to be processed into a visual processing model to obtain the visual features output by the visual processing model.
[0156] S209. If at least one feature matching value satisfies a preset feature matching value, determine the third overlapping information.
[0157] The preset feature matching value can be expressed as a number, such as 0.9, or as a percentage, such as 90%.
[0158] Understandably, multiple feature matching values are obtained by matching the visual features of the text to be processed with the features of pre-stored candidate resources. Therefore, by comparing these feature matching values with preset feature matching values, and if at least one feature matching value satisfies the preset feature matching value, it can be determined that the text to be processed contains duplicates.
[0159] S210. Combine at least one of the first overlapping information, the second overlapping information, and the third overlapping information to perform deduplication processing on the text to be processed.
[0160] The explanation of step S210 is the same as that in the above embodiments, and will not be repeated here.
[0161] S211. Control the uploading of the text to be processed.
[0162] The purpose of this step is to upload the unique text after deduplication of the text to be processed.
[0163] Understandably, by controlling the upload process of the text to be processed, we can not only ensure that the uploaded content is up-to-date, but also avoid uploading irrelevant or duplicate data, thus ensuring the effectiveness of data processing.
[0164] S212. Store the text to be processed as a new reference text.
[0165] Understandably, storing the text to be processed as the new reference text not only means that the data in the hash table can be updated, but also that the hash value corresponding to the text to be processed can be stored in the hash table, ensuring that the data in the hash table is always up-to-date.
[0166] Optionally, in addition to storing the text to be processed, this application also needs to store the visual features corresponding to the text to be processed, including: storing the visual features corresponding to the text to be processed as new visual features.
[0167] Understandably, by storing the visual features of the text to be processed as new visual features, the features of candidate resources can be updated to ensure that these features are always up-to-date, enabling more accurate resource matching and analysis.
[0168] The text deduplication method provided in this embodiment first obtains the text to be processed, then extracts at least one text tag contained in the text and concatenates it into a string to be processed. This string is then processed using a preset hash function to obtain a target hash value. The prefix of this target hash value is matched with the hash value of a reference text, and the result is used as the first overlap information. Subsequently, the text to be processed is input into an image recognition model, and candidate tags with sufficient confidence are selected as target tags. The second overlap information is determined based on the cosine similarity between the target tags and pre-stored reference texts. Simultaneously, the visual features of the text to be processed are matched with pre-stored candidate resource features. If all matching results meet preset conditions, the third overlap information is determined. Finally, at least one of the first, second, and third overlap information is combined to perform deduplication on the text to be processed. Furthermore, after deduplication, the text to be processed is uploaded and stored as a new reference text. This method comprehensively judges whether text is duplicated through multiple dimensions, which not only effectively avoids the repeated uploading of identical or similar texts but also saves storage space, thereby helping to ensure the diversity and novelty of uploaded content.
[0169] Figure 3 A schematic diagram of the text deduplication device provided in this application is shown below. Figure 3 As shown, the text deduplication device 300 provided in this embodiment includes:
[0170] Module 301 is used to acquire the text to be processed;
[0171] The determining module 302 is used to determine the first overlapping information corresponding to the text to be processed based on a pre-set hash mapping table and the hash value corresponding to at least one pre-stored reference text.
[0172] The determining module 302 is further configured to determine the second overlap information corresponding to the text to be processed based on the cosine similarity between the text to be processed and at least one pre-stored reference text;
[0173] The determining module 302 is further configured to match at least one pre-stored candidate resource feature with the visual features corresponding to the text to be processed, and determine the third overlapping information corresponding to the text to be processed;
[0174] The processing module 303 is used to perform deduplication processing on the text to be processed by combining at least one of the first overlapping information, the second overlapping information, and the third overlapping information.
[0175] In one possible implementation, the device further includes: splicing module 304;
[0176] The splicing module 304 is used to extract at least one text tag contained in the text to be processed and splice them to form the string to be processed;
[0177] The processing module 303 is also used to process the string to be processed using a preset hash function to obtain the target hash value corresponding to the text to be processed;
[0178] The device also includes: a matching module 305;
[0179] The matching module 305 is used to match the prefix of the target hash value with the hash value corresponding to the reference text, and use the matching result as the first overlap information.
[0180] In one possible implementation, the device further includes: an input module 306;
[0181] The input module 306 is used to input the text to be processed into the image recognition model to obtain multiple candidate labels output by the image recognition model, as well as the label confidence score corresponding to each candidate label;
[0182] The determination module 302 is also used to determine, for any candidate label, the candidate label as the target label if the label confidence of the candidate label meets the preset confidence.
[0183] The determination module 302 is specifically used to determine the second overlap information corresponding to the text to be processed based on the cosine similarity between the target label and at least one pre-stored reference text.
[0184] In one possible implementation, the input module 306 is further configured to input the target label into the first word vector model to obtain the first word vector of the target label;
[0185] The input module 306 is also used to input at least one pre-stored reference text into the second word vector model to obtain at least one second word vector;
[0186] The determination module 302 is specifically used to determine at least one cosine similarity based on the first word vector and at least one second word vector.
[0187] In one possible implementation, the matching module 305 is further configured to match at least one pre-stored candidate resource feature with the visual features of the text to be processed to obtain at least one feature matching value.
[0188] The determining module 302 is specifically used to determine the third overlap information when at least one feature matching value satisfies a preset feature matching value.
[0189] In one possible implementation, the processing module 303 is further configured to perform vectorization processing on at least one of the candidate resource features to obtain at least one candidate resource feature vector;
[0190] The processing module 303 is further configured to perform vectorization processing on the visual features of the text to be processed to obtain a visual feature vector.
[0191] The determining module 302 is specifically used to determine the at least one feature matching value based on the at least one candidate resource feature vector and the visual feature vector.
[0192] In one possible implementation, the device further includes: a control module 307;
[0193] Control module 307 is used to control the uploading of the text to be processed;
[0194] The determination module 302 is also used to store the text to be processed as a new reference text.
[0195] The text deduplication device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.
[0196] Figure 4 A schematic diagram of the text deduplication device provided in this application. Figure 4 As shown, the electronic device 400 provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the device 500 further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus 404.
[0197] In a specific implementation, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the above-described method.
[0198] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.
[0199] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0200] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.
[0201] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0202] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0203] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.
[0204] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.
[0205] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.
[0206] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.
[0207] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0208] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0209] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0210] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0211] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.
[0212] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0213] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.
[0214] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.
[0215] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.
[0216] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.
[0217] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.
[0218] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.
[0219] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A text deduplication method, characterized in that, The method includes: Get the text to be processed; Based on a pre-set hash mapping table and the hash value corresponding to at least one pre-stored reference text, the first overlapping information corresponding to the text to be processed is determined; Based on the cosine similarity between the text to be processed and at least one pre-stored reference text, the second overlap information corresponding to the text to be processed is determined; Match at least one pre-stored candidate resource feature with the visual features corresponding to the text to be processed to determine the third overlapping information corresponding to the text to be processed; The text to be processed is deduplicated by combining at least one of the first overlapping information, the second overlapping information, and the third overlapping information.
2. The method according to claim 1, characterized in that, The determination of the first overlap information corresponding to the text to be processed, based on a pre-set hash mapping table and the hash value corresponding to at least one pre-stored reference text, includes: Extract at least one text tag contained in the text to be processed, and concatenate them to form the string to be processed; The string to be processed is processed using a preset hash function to obtain the target hash value corresponding to the text to be processed; The prefix of the target hash value is matched with the hash value corresponding to the reference text, and the matching result is used as the first overlap information.
3. The method according to claim 1, characterized in that, The step of determining the second overlap information corresponding to the text to be processed based on the cosine similarity between the text to be processed and at least one pre-stored reference text includes: The text to be processed is input into an image recognition model to obtain multiple candidate labels output by the image recognition model, and the label confidence score corresponding to each candidate label; For any candidate label, if the label confidence of the candidate label meets the preset confidence level, the candidate label is used as the target label; The second overlap information corresponding to the text to be processed is determined based on the cosine similarity between the target label and at least one pre-stored reference text.
4. The method according to claim 3, characterized in that, Determining the cosine similarity between the target tag and at least one pre-stored reference text includes: The target label is input into the first word vector model to obtain the first word vector of the target label; Input at least one pre-stored reference text into the second word vector model to obtain at least one second word vector; At least one cosine similarity is determined based on the first word vector and at least one of the second word vectors.
5. The method according to claim 1, characterized in that, The step of matching at least one pre-stored candidate resource feature with the visual features corresponding to the text to be processed to determine the third overlap information corresponding to the text to be processed includes: Match at least one pre-stored candidate resource feature with the visual features of the text to be processed to obtain at least one feature matching value; The third overlap information is determined when at least one of the feature matching values satisfies a preset feature matching value.
6. The method according to claim 5, characterized in that, The step of matching at least one pre-stored candidate resource feature with the visual features of the text to be processed to obtain at least one feature matching value includes: At least one of the candidate resource features is vectorized to obtain at least one candidate resource feature vector; The visual features of the text to be processed are vectorized to obtain visual feature vectors; The at least one feature matching value is determined based on the at least one candidate resource feature vector and the visual feature vector.
7. The method according to claim 1, characterized in that, After performing deduplication on the text to be processed, the method further includes: Control the upload of the text to be processed; The text to be processed is stored as a new reference text.
8. A text deduplication device, characterized in that, include: The acquisition module is used to acquire the text to be processed; The determination module is used to determine the first overlap information corresponding to the text to be processed based on a pre-set hash mapping table and the hash value corresponding to at least one pre-stored reference text. The determining module is further configured to determine the second overlap information corresponding to the text to be processed based on the cosine similarity between the text to be processed and at least one pre-stored reference text; The determining module is also used to match at least one pre-stored candidate resource feature with the visual features corresponding to the text to be processed, and determine the third overlapping information corresponding to the text to be processed; The processing module is used to perform deduplication processing on the text to be processed by combining at least one of the first overlapping information, the second overlapping information, and the third overlapping information.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.
10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.