Semantic deduplication method and device for comment data, storage medium and electronic device

Abstract

This application discloses a semantic deduplication method, storage medium, and electronic device for comment data, relating to the field of artificial intelligence technology. The semantic deduplication method for comment data includes: filtering original comment data based on preset filtering rules to obtain comment data to be deduplicated, wherein the preset filtering rules are used to filter comment data that meets product relevance and authenticity; determining candidate comment data from the comment data to be deduplicated based on preset quantification rules, wherein the preset quantification rules are used to quantify and score the comment data to be deduplicated; and performing internal semantic deduplication and historical semantic deduplication on the candidate comment data to obtain deduplicated comment data. Using the above technical solution, the problem of how to efficiently filter valuable comment information from massive amounts of comment data is solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a method, apparatus, storage medium, and electronic device for semantic deduplication of comment data. Background Technology

[0002] In e-commerce platforms and social media, user reviews contain a wealth of valuable feedback for product improvement. However, data cleaning and filtering are necessary from massive amounts of reviews to extract truly valuable feedback. Existing technologies primarily employ keyword matching, sentiment lexicon analysis, topic models, or text similarity algorithms to process reviews. These methods mainly rely on word frequency statistics or string matching, failing to deeply understand the complex semantics of natural language and struggle to accurately identify the implicit user needs within reviews. They typically only remove identical duplicate content, lacking effective mechanisms for identifying and merging deeply duplicated reviews with different expressions but identical semantics. This results in a still massive data pool after initial filtering, with a large amount of homogeneous information repetitive, severely impacting the efficiency and accuracy of subsequent analysis.

[0003] Therefore, there is a technical problem in related technologies: how to efficiently filter out valuable comment information from massive amounts of comment data. Summary of the Invention

[0004] This application provides a semantic deduplication method, apparatus, storage medium, and electronic device for comment data, to at least solve the technical problem in the related art of how to efficiently filter out valuable comment information from massive comment data.

[0005] According to one embodiment of this application, a semantic deduplication method for comment data is provided, comprising: filtering original comment data based on preset filtering rules to obtain comment data to be deduplicated, wherein the preset filtering rules are used to filter comment data that meets the requirements of product relevance and authenticity; determining candidate comment data from the comment data to be deduplicated based on preset quantification rules, wherein the preset quantification rules are used to quantify and score the comment data to be deduplicated; and performing internal semantic deduplication and historical semantic deduplication on the candidate comment data to obtain deduplicated comment data.

[0006] In an exemplary embodiment, filtering original comment data based on preset filtering rules to obtain deduplicated comment data includes: filtering the original comment data using a first preset rule to obtain first comment data, wherein the first preset rule includes a comment text length rule, an advertising filtering rule, and a duplication filtering rule, wherein the length rule is used to filter comment data whose text length is less than a preset length, the advertising filtering rule is used to filter comment data containing preset advertising keywords, and the duplication filtering rule is used to filter comment data whose text is completely repeated; filtering the first comment data using a first large language model and a first prompt word to obtain second comment data, wherein the first prompt word is used to instruct the first large language model to filter the first comment data according to the second preset rule, and the second preset rule is used to filter comment data that does not meet the characteristics of a genuine comment; and determining the second comment data as the deduplicated comment data.

[0007] In an exemplary embodiment, determining candidate comment data from the deduplicated comment data based on preset quantification rules includes: quantifying and scoring the deduplicated comment data using a second language model and a second prompt word to obtain a score value for the deduplicated comment data, wherein the second prompt word is used to instruct the second language model to score the deduplicated comment data based on the preset quantification rules, the preset quantification rules including bonus rules and deduction rules, the bonus rules adding points to comment data that mentions product usage experience and improvement suggestions, and the deduction rules deducting points from comment data that mentions content other than the product itself; and identifying comment data in the deduplicated comment data with a score value greater than a preset scoring threshold as candidate comment data.

[0008] In an exemplary embodiment, performing internal semantic deduplication and historical semantic deduplication on the candidate comment data to obtain deduplicated comment data includes: performing semantic clustering on the candidate comment data to obtain multiple comment groups, and determining representative comments for each comment group to obtain multiple representative comments; determining the multiple representative comments as the internal semantic deduplication result; and performing historical semantic deduplication on the internal semantic deduplication result to obtain the deduplicated comment data.

[0009] In an exemplary embodiment, semantic clustering is performed on the candidate comment data to obtain multiple comment groups, and representative comments are determined for each comment group to obtain multiple representative comments. This includes: semantic extraction of the candidate comment data based on preset semantic extraction rules to obtain the core semantics of the candidate comment data; converting the core semantics into semantic vectors and clustering the semantic vectors based on cosine similarity to obtain multiple semantic vector clusters; determining the comment group corresponding to each semantic vector cluster based on the comment data corresponding to each semantic vector cluster to obtain the multiple comment groups; and determining the comment with the highest rating in each comment group as the representative comment of each comment group to obtain the multiple representative comments.

[0010] In an exemplary embodiment, semantic extraction is performed on the candidate review data based on preset semantic extraction rules to obtain the core semantics of the candidate review data. This includes: extracting semantics from the candidate review data using a third language model and a third prompt word to obtain product semantics, wherein the third prompt word is used to instruct the third language model to extract semantic information related to the product's attributes or user experience from the candidate review data; performing syntactic structure transformation on the product semantics to obtain a standard format semantics with a preset syntactic structure; and determining the standard format semantics as the core semantics.

[0011] In an exemplary embodiment, performing historical semantic deduplication on the internal semantic deduplication result to obtain the deduplicated comment data includes: for a first comment in the internal semantic deduplication result, if it is determined that there is a second comment in the historical deduplicated comment data with a semantic similarity greater than a preset similarity to the first comment, obtaining a first rating value of the first comment and a second rating value of the second comment; if it is determined that the first rating value is less than the second rating value, deleting the first comment from the internal semantic deduplication result to obtain the deduplicated comment data.

[0012] According to another aspect of the embodiments of this application, a semantic deduplication device for comment data is also provided, comprising: a filtering module, configured to filter original comment data based on preset filtering rules to obtain comment data to be deduplicated, wherein the preset filtering rules are used to filter comment data that meets the requirements of product relevance and authenticity; a determining module, configured to determine candidate comment data from the comment data to be deduplicated based on preset quantification rules, wherein the preset quantification rules are used to quantify and score the comment data to be deduplicated; and a deduplication module, configured to perform internal semantic deduplication and historical semantic deduplication on the candidate comment data to obtain deduplicated comment data.

[0013] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the semantic deduplication method for the above-mentioned comment data at runtime.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the semantic deduplication method for the comment data described above through the computer program.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0016] In this embodiment, real product-related reviews are filtered from the original review data based on preset filtering rules to obtain deduplicated review data. Then, the deduplicated review data is quantitatively scored based on preset quantification rules to evaluate the value of each review, thereby identifying high-value candidate reviews from the deduplicated review data. Finally, internal semantic deduplication and historical semantic deduplication are performed on the candidate review data, retaining the highest-value reviews from those with the same or similar semantics, thus obtaining deduplicated review data. This solves the technical problem of how to efficiently filter valuable review information from massive amounts of review data, thereby achieving the effect of efficiently filtering valuable review information from massive amounts of review data. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the hardware environment for a semantic deduplication method for comment data according to an embodiment of this application;

[0020] Figure 2 This is a flowchart of a semantic deduplication method for comment data according to an embodiment of this application;

[0021] Figure 3 This is a structural block diagram of a comment selection system according to an optional embodiment of this application;

[0022] Figure 4 This is a structural block diagram of a semantic deduplication device for comment data according to an embodiment of this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] According to one aspect of the embodiments of this application, a semantic deduplication method for comment data is provided. This semantic deduplication method for comment data is widely applied to digital control scenarios such as smart homes, smart home ecosystems, intelligence house ecosystems, whole-house intelligence, digital homes, networked homes, automated homes, electronic homes, digital homes, and digital homes. It relies on technologies such as the Internet of Things (IoT), cloud computing, artificial intelligence (AI), digital technology, digital twins, and AR / VR to build intelligent, automated, personalized, and customized digital ecosystems. Optionally, in this embodiment, the above-mentioned semantic deduplication method for comment data can be applied to, for example... Figure 1 The hardware environment shown consists of terminal device 102 and server 104. For example... Figure 1As shown, server 104 is connected to terminal device 102 via a network and can be used to provide services (such as application services) to the terminal or clients installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0026] The aforementioned network communication architecture includes, but is not limited to, two types of links: wired transmission and wireless connection. Among them, the wired network can cover wide area networks, metropolitan area networks, and local area networks, while the wireless network supports communication standards such as WIFI, Bluetooth, and 5G. It can adapt to the digital technology transmission needs of the Internet of Things (IoT), the Internet of Things, and the Internet of Things blockchain, ensuring stable interaction of digital information, enabling devices to make autonomous, accurate, precise, and accurate judgments and adjustments, and supporting personalized and customized intelligent capabilities such as voice interaction, remote control, and self-control. Terminal devices 102 include, but are not limited to, PCs, mobile phones, tablets, and all categories of digital home appliances, digital electrical appliances, smart home appliances, and whole-house smart devices. Specifically, they include: smart air conditioners, air conditioning equipment, smart range hoods, smart refrigerators, smart ovens, grilling and cooking equipment, smart stoves, smart washing machines, washing, rinsing, drying and garment processing equipment, smart water heaters, smart washing equipment, smart dishwashers, smart projectors, smart TVs, smart clothes racks, smart curtains, smart audio-visual equipment, smart sockets, smart speakers, smart speakers, smart fresh air systems, smart kitchen and bathroom systems, kitchens, cabinets, kitchen cabinet equipment, smart bathroom equipment, robot vacuum cleaners, robot window cleaners, robot mopping, air purifiers, smart steam ovens, smart microwave ovens, smart water heaters, smart air purifiers, smart water dispensers, smart door locks, smart lighting fixtures, smart light bulbs, smart air detectors, smart remote controls, and smart gateways. They also cover smart terminals for security, lighting, and other digital home furnishings.

[0027] This embodiment provides a semantic deduplication method for comment data. Figure 2 This is a flowchart of a semantic deduplication method for comment data according to an embodiment of this application. The process includes the following steps:

[0028] Step S202: Filter the original comment data based on preset filtering rules to obtain duplicate comment data to be deduplicated, wherein the preset filtering rules are used to filter comment data that meet the requirements of product relevance and authenticity;

[0029] Step S204: Determine candidate comment data from the deduplicated comment data based on preset quantization rules, wherein the preset quantization rules are used to quantify and score the deduplicated comment data;

[0030] Step S206: Perform internal semantic deduplication and historical semantic deduplication on the candidate comment data to obtain deduplicated comment data.

[0031] Optionally, in the above steps, the original comment data is initially screened using preset filtering rules to remove comments that are irrelevant to the product or lack authenticity. For example, advertisements, meaningless spam, and comments that are too short are filtered out, thus obtaining deduplicated comment data that meets the requirements of product relevance and authenticity. Next, the deduplicated comment data is quantitatively scored based on preset quantification rules to measure the value of each comment in expressing user needs. For example, scores are given based on the level of detail, whether it contains improvement suggestions, and whether it is a general evaluation. Comments with higher scores are identified as candidate comment data. Finally, the candidate comment data undergoes internal semantic deduplication and historical semantic deduplication. That is, the comments within the current batch are first deduplicated through semantic clustering, and then semantic similarity is compared with historically retained comments to remove comments with duplicate expressions. Finally, concise and information-dense deduplicated comment data is obtained. Through the above method, comments with practical value can be efficiently screened from massive amounts of comments, and redundant information at the semantic level can be effectively eliminated, improving the accuracy and efficiency of subsequent analysis.

[0032] Through the above steps, authentic product-related reviews are filtered from the original review data based on preset screening rules to obtain deduplicated review data. Then, based on preset quantification rules, each review is quantitatively scored to evaluate its value, thereby identifying high-value candidate reviews. Finally, internal and historical semantic deduplication is performed on the candidate reviews, retaining the highest-value reviews from those with similar or identical semantics, resulting in deduplicated review data. This solves the technical problem of efficiently filtering valuable review information from massive amounts of review data, thus achieving the effect of efficiently filtering valuable review information from massive amounts of review data.

[0033] In an exemplary embodiment, filtering original comment data based on preset filtering rules to obtain deduplicated comment data includes: filtering the original comment data using a first preset rule to obtain first comment data, wherein the first preset rule includes a comment text length rule, an advertising filtering rule, and a duplication filtering rule, wherein the length rule is used to filter comment data whose text length is less than a preset length, the advertising filtering rule is used to filter comment data containing preset advertising keywords, and the duplication filtering rule is used to filter comment data whose text is completely repeated; filtering the first comment data using a first large language model and a first prompt word to obtain second comment data, wherein the first prompt word is used to instruct the first large language model to filter the first comment data according to the second preset rule, and the second preset rule is used to filter comment data that does not meet the characteristics of a genuine comment; and determining the second comment data as the deduplicated comment data.

[0034] Optionally, in the above embodiments, the preset filtering rules are implemented through a two-stage filtering process. The first stage uses a first preset rule to quickly filter the original comment data. The length rule filters comments with text lengths shorter than a preset length (e.g., less than 4 characters), the advertising filtering rule filters comments containing preset advertising keywords (e.g., "recommend to everyone," "product recommendation expert," etc.) or containing a large number of emoticons, and the duplication filtering rule filters duplicate comments with identical text, thus obtaining the first comment data. The second stage uses a first large language model combined with a first prompt word to perform deep filtering on the first comment data. The first prompt word instructs the first large language model to determine whether a comment is a genuine user comment based on the second preset rule. The second preset rule includes features such as whether the comment lists multiple common marketing terms, whether it uses absolute and exaggerated expressions, and whether it is always described in the third person. The large language model outputs recognition results based on the above rules, filtering out comments that do not conform to the characteristics of genuine user comments, obtaining the second comment data, and identifying it as the comment data to be deduplicated. Through this two-stage filtering, while ensuring filtering efficiency, the accuracy of identifying the authenticity and relevance of comments can be effectively improved, providing a high-quality data foundation for subsequent processing.

[0035] In an exemplary embodiment, determining candidate comment data from the deduplicated comment data based on preset quantification rules includes: quantifying and scoring the deduplicated comment data using a second language model and a second prompt word to obtain a score value for the deduplicated comment data, wherein the second prompt word is used to instruct the second language model to score the deduplicated comment data based on the preset quantification rules, the preset quantification rules including bonus rules and deduction rules, the bonus rules adding points to comment data that mentions product usage experience and improvement suggestions, and the deduction rules deducting points from comment data that mentions content other than the product itself; and identifying comment data in the deduplicated comment data with a score value greater than a preset scoring threshold as candidate comment data.

[0036] Optionally, in the above embodiments, the preset quantification rules achieve quantitative scoring of the deduplicated comment data through a second language model combined with second prompt words. The second prompt words instruct the second language model to score each comment according to predefined bonus and deduction rules. Bonus rules include: the comment describes the product's impact in a specific usage scenario; it clearly expresses suggestions for improving the product's function or appearance; it includes actionable content such as detailed usage steps or comparative experiences. Deduction rules include: the comment uses only broad, subjective vocabulary without detailed support; it mainly mentions non-product-related content such as price, customer service, logistics, and installation. The second language model outputs the rule number and number of hits according to the above rules. The system calculates the comment's score based on the preset score value corresponding to each rule. Comment data with scores greater than a preset scoring threshold are identified as candidate comment data, thereby filtering out comments with clear user demand value. Through this quantitative scoring mechanism, the value assessment of comment text can be transformed into comparable numerical indicators, achieving accurate screening of comment value.

[0037] In an exemplary embodiment, performing internal semantic deduplication and historical semantic deduplication on the candidate comment data to obtain deduplicated comment data includes: performing semantic clustering on the candidate comment data to obtain multiple comment groups, and determining representative comments for each comment group to obtain multiple representative comments; determining the multiple representative comments as the internal semantic deduplication result; and performing historical semantic deduplication on the internal semantic deduplication result to obtain the deduplicated comment data.

[0038] Optionally, in the above embodiments, internal semantic deduplication and historical semantic deduplication employ a hierarchical processing mechanism. First, the candidate comment data is semantically clustered, grouping semantically similar comments into the same comment group. Within each comment group, the most representative comment (the highest-rated comment) is selected as the representative comment for that group, resulting in multiple representative comments. These representative comments constitute the internal semantic deduplication result. Subsequently, historical semantic deduplication is performed on this internal semantic deduplication result. This involves comparing the semantic similarity of each representative comment in the internal semantic deduplication result with previously retained comments. If the semantic similarity exceeds a preset threshold, the comment with the higher rating is retained, and the other is deleted, ultimately yielding the deduplicated comment data. Through this hierarchical deduplication mechanism, both internal simplification of the current batch of comment data is achieved, while ensuring global consistency between new and historical comments. This effectively prevents the long-term accumulation of comments with the same semantics in the data pool, significantly improving the information density of the data.

[0039] In an exemplary embodiment, semantic clustering is performed on the candidate comment data to obtain multiple comment groups, and representative comments are determined for each comment group to obtain multiple representative comments. This includes: semantic extraction of the candidate comment data based on preset semantic extraction rules to obtain the core semantics of the candidate comment data; converting the core semantics into semantic vectors and clustering the semantic vectors based on cosine similarity to obtain multiple semantic vector clusters; determining the comment group corresponding to each semantic vector cluster based on the comment data corresponding to each semantic vector cluster to obtain the multiple comment groups; and determining the comment with the highest rating in each comment group as the representative comment of each comment group to obtain the multiple representative comments.

[0040] Optionally, in the above embodiments, semantic clustering is implemented through semantic extraction, vectorization, and clustering algorithms. First, semantic extraction is performed on the candidate comment data based on preset semantic extraction rules to obtain the core semantics of each comment. This core semantics adopts a standardized and structured expression form; for example, "This panel is easy to clean" is extracted as "Need easy-to-clean panels." Next, a semantic vector model is used to convert the core semantics into semantic vectors, and these vectors are clustered based on cosine similarity. For example, agglomerative hierarchical clustering algorithm is used to group vectors with cosine similarity higher than a preset threshold into the same cluster, resulting in multiple semantic vector clusters. Each semantic vector cluster corresponds to a comment group, where comments express the same or highly similar user needs at the semantic level. In each comment group, the comment with the highest rating is selected as the representative comment for that group, ultimately resulting in multiple representative comments. Through the above semantic clustering method, comments with the same semantics but different expressions can be accurately identified, and the most valuable comment is retained within each semantic group, thereby achieving efficient semantic deduplication.

[0041] In an exemplary embodiment, semantic extraction is performed on the candidate review data based on preset semantic extraction rules to obtain the core semantics of the candidate review data. This includes: extracting semantics from the candidate review data using a third language model and a third prompt word to obtain product semantics, wherein the third prompt word is used to instruct the third language model to extract semantic information related to the product's attributes or user experience from the candidate review data; performing syntactic structure transformation on the product semantics to obtain a standard format semantics with a preset syntactic structure; and determining the standard format semantics as the core semantics.

[0042] Optionally, in the above embodiments, the preset semantic extraction rules utilize a third language model combined with third prompt words to achieve accurate semantic extraction and standardization. The third prompt words instruct the third language model to extract semantic information directly related to the product's specific functions, operation process, performance, objective appearance attributes, and user scenarios from the candidate review data, excluding non-core product aspects such as quality, cost-effectiveness, logistics, customer service, and installation, thus obtaining the product semantics. Simultaneously, the third prompt words also require the language model to convert the extracted product semantics into a standard format semantics with a preset grammatical structure, such as uniformly adopting the expression form of "verb + entity noun," making semantics with the same meaning consistent in text expression. The standard format semantics obtained after conversion are determined as the core semantics. Through the above method, not only can the deep semantics related to the product in the reviews be accurately extracted, but the standardized expression can also provide high-quality input for subsequent semantic vectorization and clustering deduplication, significantly improving the accuracy and consistency of semantic deduplication.

[0043] In an exemplary embodiment, performing historical semantic deduplication on the internal semantic deduplication result to obtain the deduplicated comment data includes: for a first comment in the internal semantic deduplication result, if it is determined that there is a second comment in the historical deduplicated comment data with a semantic similarity greater than a preset similarity to the first comment, obtaining a first rating value of the first comment and a second rating value of the second comment; if it is determined that the first rating value is less than the second rating value, deleting the first comment from the internal semantic deduplication result to obtain the deduplicated comment data.

[0044] Optionally, in the above embodiments, historical semantic deduplication is achieved through a line-by-line comparison and scoring optimization. For each representative comment in the internal semantic deduplication result (denoted as the first comment), its semantic vector is compared with the semantic vectors of each comment in the historical deduplicated comment data using cosine similarity. When the semantic similarity between a historical comment (denoted as the second comment) and the first comment is greater than a preset similarity, it indicates that the two comments are substantially duplicated at the semantic level. At this time, the first score of the first comment and the second score of the second comment are obtained and compared. If the first score is less than the second score, it means that the value of the first comment is lower than that of the retained historical comment, so the first comment is deleted from the internal semantic deduplication result; otherwise, if the first score is greater than the second score, the first comment is retained, and the second comment is removed from the historical deduplicated comment data accordingly. Finally, the updated deduplicated comment data is obtained. Through the above historical semantic deduplication mechanism, the state of retaining only one highest-scoring comment with the same semantics can be maintained globally, effectively avoiding redundancy of effective information as data accumulates, and ensuring the simplicity and optimality of the data pool.

[0045] In an optional embodiment, a comment selection system based on LLM and semantic deduplication is provided, such as Figure 3 As shown, the system includes a data acquisition and preprocessing module, an LLM-based semantic understanding and scoring module, a multi-layer deduplication module, and a data output module. The following section, in conjunction with a specific application scenario, details the complete processing flow of the system according to the above module sequence.

[0046] I. Data Acquisition and Preprocessing Module.

[0047] The data acquisition and preprocessing module is used to obtain raw comment data from data sources and perform preliminary cleaning and filtering based on the data source and preset filtering rules. In this embodiment, the system first obtains the raw comment data newly added that day from the ELASTIC SEARCH database and parses the structured fields such as industry and category of the comments. Depending on the data source, the system adopts differentiated preprocessing strategies for e-commerce data and social media data.

[0048] For e-commerce data, the system first filters out comments with less than four characters in length, and then deduplicates comments with identical text. For the remaining comments, the system matches the comment text with a preset list of brands and categories, extracts the brands (such as "Brand A" and "Brand B") and categories (such as "washing machine" and "microwave oven") mentioned in the comments, and determines the corresponding industry (such as "washing machine" and "kitchen appliance") based on a predefined category-industry mapping table.

[0049] For social media data, the system also first filters out comments shorter than four characters. Since a high percentage of invalid content, such as advertisements and reviews, is found in social media data, the system first performs initial screening using keyword matching: comments containing common advertising keywords (such as "home renovation hard-earned experience," "feedback," "recommendations," "product recommendation expert") or more than five emojis are filtered out. Subsequently, the remaining comments are cleaned, removing "#topics," "@users," and other similar content from the original text, and then deduplicating comments with identical text.

[0050] After the initial cleaning described above, the system further uses a large language model to identify genuine user comments on the social media data. The system constructs the following prompt word template and sends it to the large language model:

[0051] "If a review contains at least four common marketing terms, such as 'ultra-thin and embeddable,' 'extra-large French-style storage space,' 'multi-temperature zone design,' or 'intelligent control system,' it is considered an advertisement. If a review claims that the product is perfect in all aspects, including appearance, space, energy efficiency, functionality, and intelligent control, and does not mention any inconveniences, shortcomings, or suggestions for improvement, it is considered an advertisement. If a review uses non-colloquial expressions, employs absolute phrases such as 'whether it' or 'it can', and the positive emotions are extremely strong and exaggerated, it is considered an advertisement. If the review does not use the first-person pronoun 'I' and is always described in the third person, it is considered an advertisement. Even if the review uses the first person, if it meets multiple of the aforementioned characteristics, it is still considered an advertisement."

[0052] The large language model outputs a judgment result based on the above rules, and the system filters out data identified as non-genuine user reviews. For the remaining reviews, the system performs the same brand, category, and industry parsing processing as the e-commerce data to obtain the data for deduplicating reviews.

[0053] Through the above two-stage screening, the system identifies review data that meets the requirements of product relevance and authenticity as the data to be deduplicated, providing a high-quality data foundation for subsequent processing.

[0054] II. Semantic Understanding and Scoring Module Based on LLM.

[0055] The LLM-based semantic understanding and scoring module is used to perform semantic extraction and quantitative scoring on deduplicated comment data. This module includes a semantic understanding and coarse screening unit, and a value assessment and fine screening unit.

[0056] (a) Semantic understanding and coarse screening units.

[0057] In this embodiment, the semantic understanding and coarse screening unit uses a large language model combined with prompt words to extract semantic information related to product attributes or user experience from the deduplicated review data and convert it into core semantics in a standardized format.

[0058] The system uses prompt words based on dynamic parameter templates to guide a large language model in extracting the core semantics of comments. The prompt word template is as follows:

[0059] "You are a highly rigorous user review semantic extractor. You need to transform real user reviews into semantically equivalent expressions in a standard format. Please fully understand the entire semantics of the review, and then transform all eligible semantics into semantically equivalent expressions in the format of verb + entity noun. Please extract only semantics directly related to the specific function, operation process, performance, objective appearance attributes (such as color, special design), and user scenario of {p}. Exclude the following irrelevant aspects: product quality, cost-effectiveness, logistics and delivery, customer service, installation, accessories, and quality control. Ensure that the extracted semantics focus on a specific point of the product so that they can be transformed into clear product optimization directions later." If the evaluation of {basic_feature} contains only vague expressions such as 'good' or 'not bad' without specific details, such as 'good cooling effect', please do not extract it. If the comment cannot be converted into semantics that meet the requirements, its semantics are []. User's feelings about the special features of {p}, such as: {advanced_feature}, mentioning whether the product or its components are easy to clean and maintain, semantic example: This panel is easy to clean - Need an easy-to-clean panel. User's interactive experience of {p}, such as {interact_feature}, semantic example: Can remotely control the air conditioner - Remotely control the air conditioner.

[0060] In the template above, {p} dynamically represents the product category mentioned in the current comment, {basic_feature} represents the basic function of that category, {advanced_feature} represents advanced functions, and {interact_feature} represents pre-summarized interactive functions. The large language model extracts product semantics directly related to product functions, performance, appearance, and usage scenarios from the comments as required, and converts them into standard semantic formats of "verb + entity noun." For example, for the comment "This panel is easy to clean," the system extracts and converts it into "Needs an easy-to-clean panel." For comments from which no effective semantics can be extracted, the system directly filters them, completing the initial screening.

[0061] (ii) Value assessment and screening unit.

[0062] The value assessment and fine screening unit uses a large language model combined with another set of prompt words to quantify and score the comments after the initial screening, and selects candidate comments with higher scores.

[0063] The system uses the following prompts:

[0064] "You are a strict and impartial user review scorer. Your task is to analyze whether the reviews contain valuable and specific descriptions, and to penalize empty and vague statements."

[0065] 1. Usage Scenarios and Impacts: The review describes how the product affects specific groups of people, pets (e.g., it is safe for children, improves sleep quality, and was bought to make baby formula), how it affects the environment (white color and overall color scheme), or in what specific life situations it is used (specific environment / background).

[0066] 2. Pain Points and Suggestions: The review clearly expresses concerns, dissatisfaction, or specific suggestions for improvement regarding a particular issue with the product (must be related to functionality, appearance, performance, or whether it consumes a lot of electricity or water).

[0067] 3. Operability and Detail: Reviews should include detailed usage steps (e.g., 'During installation, flush with water for 10 minutes until the TDS value stabilizes below 20 to ensure successful activation.') or comparative experiences (comparing only product functions and appearance; comparisons of quality, price, etc., are not considered. For example, 'Compared to the XX brand I used before, this one has more than twice the water output, filling a kettle almost 30 seconds faster.'). Avoid reviews lacking detailed process descriptions, such as: 'Installation was very simple according to the instructions.' (without explaining how simple it is).

[0068] 4. Subjective and general evaluations: Reviews use only broad, subjective terms without supporting details. Examples: good, very good, not bad, positive review, satisfied, convenient, easy to use, cost-effective, just right, suitable, good-looking, fully functional, easy to operate, good value for money.

[0069] 5. Service and Price: Reviews should primarily focus on the purchase process rather than the product itself. Examples: price, customer service, logistics, installation, free gifts, value for money, warranty, and after-sales service.

[0070] Operating procedures:

[0071] Each rule can be scored repeatedly, counting as many times as it is hit. Prioritize identifying bonus items (rules 1-3) to find specific value points; then identify deduction items (rules 4-5) to filter out empty content. Only output the rule number and the number of hits, in the format: rule number: number of hits, example: 1:1, 2:1, 3:2, 5:1.

[0072] The large language model outputs the rule number and the number of times each comment is matched according to the above rules, such as "1:1, 2:1, 3:2, 5:1". The system pre-sets a score for each rule; for example, matching rule 1 once earns 97 points, matching rule 2 once earns 300 points, matching rule 3 once earns 200 points, matching rule 4 once deducts 25 points, and matching rule 5 once deducts 10 points. The system calculates the total score for each comment based on the matched rules and the number of times. If the total score is lower than a preset threshold (e.g., 50 points), the comment is considered to be vague or contains too much irrelevant information and is filtered out. The remaining comments are the candidate comment data.

[0073] Through the two-stage processing of semantic understanding and coarse screening, and value assessment and fine screening, the system achieves accurate quantitative assessment of the value of comments, providing high-quality input for subsequent deduplication.

[0074] III. Multi-layer deduplication module.

[0075] The multi-layer deduplication module is used to perform semantic deduplication on candidate comment data. This module includes a semantic vector-based similarity calculation and clustering unit, an intraday deduplication unit, and a global deduplication unit.

[0076] (i) Similarity calculation and clustering units and intraday deduplication units based on semantic vectors.

[0077] The intraday deduplication unit is used to perform internal semantic deduplication on the candidate comment data for the current day.

[0078] In this embodiment, the system first uses the semantic vector model BGE (BAAI General Embedding) to vectorize the core semantics of each candidate comment. The standard format semantic text of each comment is input into the BGE model, and the model outputs the corresponding semantic vector.

[0079] For a batch of semantic vectors of the same category, the system uses an agglomerative hierarchical clustering algorithm for clustering: initially, each vector is treated as an independent cluster, and then the clusters with the highest cosine similarity are gradually merged. After merging, the average value of all vectors within the merged cluster is used as the new vector representation of that cluster. The system sets the cosine similarity threshold to a value between 0.85 and 0.95 (e.g., 0.9). When the cosine similarity of two clusters is higher than this threshold, they are merged, and clustering stops when the similarity between all clusters to be merged is lower than the threshold. Through the above clustering operation, the system obtains multiple semantic vector clusters, each corresponding to a comment group. The comments within this group express the same or highly similar user needs at the semantic level.

[0080] Within each comment group, the system selects the comment with the highest rating as the representative comment and marks its "Whether to deduplicate the representative comment" field as True, while marking other comments within the cluster as False. Simultaneously, the system stores the refined comments of the day in an Elasticsearch vector library, storing fields including comment ID, comment content, comment semantics, comment score, comment semantic vector, category, industry, whether to deduplicate the representative comment, and processing time. Through this intraday deduplication unit, the system achieves internal simplification of the day's data.

[0081] (ii) Global deduplication unit.

[0082] The global deduplication unit is used to compare the intraday deduplication results with the historical deduplication results to achieve semantic deduplication at the global level.

[0083] The system retrieves comments from the Elasticsearch database whose "Whether to deduplicate representative comments" field is True. These comments include fields such as comment ID, comment semantic vector, and comment score. For each representative comment obtained after deduplication within the day, the system calculates the cosine similarity between its semantic vector and the semantic vector of each historical representative comment.

[0084] If the cosine similarity of two comments is higher than a preset similarity threshold (e.g., 0.8), it indicates that they are semantically redundant. In this case, the system compares the ratings of the two comments and changes the "Whether to deduplicate representative comment" field of the comment with the lower rating to False, ensuring that only the highest-rated comment among multiple comments with the same semantics remains the representative comment. Through this global deduplication unit, the system maintains the state of retaining only one highest-rated comment with the same semantics globally, effectively avoiding redundancy of useful information as data accumulates and ensuring the simplicity and optimality of the data pool.

[0085] IV. Data Output Module.

[0086] The data output module is used to output the processed comment data to a designated storage for use by downstream systems.

[0087] In this embodiment, the system outputs all comments retained after daily filtering (including representative comments where the "Whether to deduplicate representative comments" field is True and non-representative comments where it is False), as well as historical comments whose status has been modified due to global deduplication, to a designated storage location. Output fields include comment content, comment semantics, comment score, comment semantic vector, category, industry, whether to deduplicate representative comments, and processing time. The output data can be consumed by downstream systems via a Kafka message queue or directly stored in an Elasticsearch database for subsequent retrieval and analysis.

[0088] The comment optimization system provided in this embodiment achieves efficient filtering and information simplification of massive comment data through the coordinated operation of a data acquisition and preprocessing module, an LLM-based semantic understanding and scoring module, a multi-layer deduplication module, and a data output module. Specifically, the system can process hundreds of thousands of comment data within tens of minutes, enabling large-scale, routine user demand insights. Simultaneously, the system effectively eliminates semantic redundancy through a two-layer mechanism of intraday deduplication and global deduplication units, significantly improving data information density. Furthermore, the system architecture has good scalability and can be flexibly applied to various data sources such as e-commerce reviews and social networks, as well as data processing scenarios across different industries and categories, demonstrating high technological reusability.

[0089] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0090] Figure 4 This is a structural block diagram of a semantic deduplication device for comment data according to an embodiment of this application; as shown below. Figure 4 As shown, it includes:

[0091] The filtering module 42 is used to filter the original comment data based on preset filtering rules to obtain duplicate comment data to be deduplicated. The preset filtering rules are used to filter comment data that meet the requirements of product relevance and authenticity.

[0092] The determining module 44 is used to determine candidate comment data from the comment data to be deduplicated based on a preset quantification rule, wherein the preset quantification rule is used to quantify and score the comment data to be deduplicated.

[0093] The deduplication module 46 is used to perform internal semantic deduplication and historical semantic deduplication on the candidate comment data to obtain deduplicated comment data.

[0094] The aforementioned device filters genuine product-related reviews from the original review data based on preset filtering rules to obtain deduplicated review data. Then, it quantifies and scores each review based on preset quantification rules to evaluate its value, thereby identifying high-value candidate reviews. Finally, it performs internal and historical semantic deduplication on the candidate reviews, retaining the highest-value reviews from those with similar or identical semantics to obtain the deduplicated review data. This solves the technical problem of efficiently filtering valuable review information from massive amounts of review data, thus achieving the effect of efficiently filtering valuable review information from massive amounts of review data.

[0095] In an exemplary embodiment, the filtering module 42 is configured to filter the original comment data using a first preset rule to obtain first comment data, wherein the first preset rule includes a comment text length rule, an advertising filtering rule, and a duplicate filtering rule. The length rule is used to filter comment data whose text length is less than a preset length, the advertising filtering rule is used to filter comment data that includes preset advertising keywords, and the duplicate filtering rule is used to filter comment data whose text is completely repeated. The first comment data is then filtered using a first large language model and a first prompt word to obtain second comment data, wherein the first prompt word is used to instruct the first large language model to filter the first comment data according to the second preset rule, and the second preset rule is used to filter comment data that does not meet the characteristics of a genuine comment. The second comment data is then determined as the comment data to be deduplicated.

[0096] In an exemplary embodiment, the determining module 44 is configured to quantify and score the deduplicated comment data using a second language model and a second prompt word to obtain a score value for the deduplicated comment data. The second prompt word instructs the second language model to score the deduplicated comment data based on a preset quantification rule. The preset quantification rule includes bonus rules and deduction rules. The bonus rules award points to comment data mentioning product usage experience and improvement suggestions, while the deduction rules deduct points from comment data mentioning content unrelated to the product itself. Comment data with a score value greater than a preset scoring threshold in the deduplicated comment data are identified as candidate comment data.

[0097] In an exemplary embodiment, the deduplication module 46 is used to perform semantic clustering on the candidate comment data to obtain multiple comment groups, and to determine the representative comments of each comment group to obtain multiple representative comments; to determine the multiple representative comments as internal semantic deduplication results; and to perform historical semantic deduplication on the internal semantic deduplication results to obtain the deduplicated comment data.

[0098] In an exemplary embodiment, the deduplication module 46 is used to perform semantic extraction on the candidate comment data based on a preset semantic extraction rule to obtain the core semantics of the candidate comment data; convert the core semantics into semantic vectors, and cluster the semantic vectors based on cosine similarity to obtain multiple semantic vector clusters; determine the comment group corresponding to each semantic vector cluster according to the comment data corresponding to each semantic vector cluster to obtain the multiple comment groups; and determine the comment with the highest rating in each comment group as the representative comment of each comment group to obtain the multiple representative comments.

[0099] In an exemplary embodiment, the deduplication module 46 is used to extract semantics from the candidate review data using a third language model and a third prompt word to obtain product semantics, wherein the third prompt word is used to instruct the third language model to extract semantic information related to the product's attributes or user experience from the candidate review data; the product semantics are then subjected to grammatical structure transformation to obtain a standard format semantics with a preset grammatical structure; and the standard format semantics are determined as the core semantics.

[0100] In an exemplary embodiment, the deduplication module 46 is configured to, for a first comment in the internal semantic deduplication result, if it is determined that there is a second comment in the historical deduplication comment data with a semantic similarity greater than a preset similarity to the first comment, obtain a first rating value of the first comment and a second rating value of the second comment; if it is determined that the first rating value is less than the second rating value, delete the first comment from the internal semantic deduplication result to obtain the deduplicated comment data.

[0101] Embodiments of this application also provide a storage medium including a stored program, wherein the program executes any of the methods described above when it is run.

[0102] Optionally, in this embodiment, the storage medium may be configured to store program code for performing the following steps:

[0103] S1, the original comment data is filtered based on preset filtering rules to obtain duplicate comment data to be deduplicated, wherein the preset filtering rules are used to filter comment data that meet the requirements of product relevance and authenticity;

[0104] S2, candidate comment data are determined from the comment data to be deduplicated based on preset quantification rules, wherein the preset quantification rules are used to quantify and score the comment data to be deduplicated;

[0105] S3, perform internal semantic deduplication and historical semantic deduplication on the candidate comment data to obtain deduplicated comment data.

[0106] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0107] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0108] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0109] S1, the original comment data is filtered based on preset filtering rules to obtain duplicate comment data to be deduplicated, wherein the preset filtering rules are used to filter comment data that meet the requirements of product relevance and authenticity;

[0110] S2, candidate comment data are determined from the comment data to be deduplicated based on preset quantification rules, wherein the preset quantification rules are used to quantify and score the comment data to be deduplicated;

[0111] S3, perform internal semantic deduplication and historical semantic deduplication on the candidate comment data to obtain deduplicated comment data.

[0112] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0113] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0114] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0115] The embodiments described herein also provide a computer program that includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the above method embodiments.

[0116] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0117] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0118] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A semantic deduplication method for comment data, characterized in that, include: The original comment data is filtered based on preset filtering rules to obtain duplicate comment data to be deduplicated. The preset filtering rules are used to filter comment data that meet the requirements of product relevance and authenticity. Candidate comment data is determined from the deduplicated comment data based on preset quantification rules, wherein the preset quantification rules are used to quantify and score the deduplicated comment data. The candidate comment data is subjected to internal semantic deduplication and historical semantic deduplication to obtain deduplicated comment data.

2. The semantic deduplication method for comment data according to claim 1, characterized in that, The original comment data is filtered based on preset filtering rules to obtain the comment data to be deduplicated, including: The original comment data is filtered by a first preset rule to obtain first comment data. The first preset rule includes a comment text length rule, an advertising filtering rule, and a duplicate filtering rule. The length rule is used to filter comment data whose text length is less than a preset length. The advertising filtering rule is used to filter comment data that includes preset advertising keywords. The duplicate filtering rule is used to filter comment data whose text is completely repeated. The first comment data is filtered using a first language model and a first prompt word to obtain second comment data. The first prompt word is used to instruct the first language model to filter the first comment data according to a second preset rule. The second preset rule is used to filter comment data that does not meet the characteristics of a real comment. The second comment data is determined as the comment data to be deduplicated.

3. The semantic deduplication method for comment data according to claim 1, characterized in that, Candidate comment data is determined from the deduplicated comment data based on preset quantization rules, including: The second language model and the second prompt word are used to quantify and score the deduplicated comment data to obtain a score value for the deduplicated comment data. The second prompt word is used to instruct the second language model to score the deduplicated comment data based on the preset quantification rules. The preset quantification rules include bonus rules and deduction rules. The bonus rules add points to comment data that mention product usage experience and improvement suggestions, and the deduction rules deduct points from comment data that mention content that is not related to the product itself. Comments with a score greater than a preset score threshold in the deduplication data are identified as candidate comment data.

4. The semantic deduplication method for comment data according to claim 3, characterized in that, The candidate comment data is subjected to internal semantic deduplication and historical semantic deduplication to obtain deduplicated comment data, including: Semantic clustering is performed on the candidate comment data to obtain multiple comment groups, and representative comments are determined for each comment group to obtain multiple representative comments; The multiple representative comments are identified as the internal semantic deduplication results; The internal semantic deduplication result is subjected to historical semantic deduplication to obtain the deduplicated comment data.

5. The semantic deduplication method for comment data according to claim 4, characterized in that, Semantic clustering is performed on the candidate comment data to obtain multiple comment groups, and representative comments are determined for each comment group to obtain multiple representative comments, including: Based on preset semantic extraction rules, semantic extraction is performed on the candidate comment data to obtain the core semantics of the candidate comment data; The core semantics are transformed into semantic vectors, and the semantic vectors are clustered based on cosine similarity to obtain multiple semantic vector clusters; Based on the comment data corresponding to each semantic vector cluster in the plurality of semantic vector clusters, the comment group corresponding to each semantic vector cluster is determined, and the plurality of comment groups are obtained; The comment with the highest rating in each comment group is identified as the representative comment for that comment group, thus obtaining the multiple representative comments.

6. The semantic deduplication method for comment data according to claim 5, characterized in that, Semantic extraction is performed on the candidate comment data based on preset semantic extraction rules to obtain the core semantics of the candidate comment data, including: The candidate review data is semantically extracted using a third language model and a third cue word to obtain product semantics. The third cue word is used to instruct the third language model to extract semantic information related to the product's attributes or user experience from the candidate review data. The semantics of the product are transformed by syntax to obtain a standard format semantics with a preset syntax structure; The standard format semantics are defined as the core semantics.

7. The semantic deduplication method for comment data according to claim 4, characterized in that, The internal semantic deduplication result is subjected to historical semantic deduplication to obtain the deduplicated comment data, including: For the first comment in the internal semantic deduplication result, if it is determined that there is a second comment in the historical deduplication comment data with a semantic similarity greater than a preset similarity to the first comment, the first rating value of the first comment and the second rating value of the second comment are obtained; If the first rating value is determined to be less than the second rating value, the first comment is deleted from the internal semantic deduplication result to obtain the deduplicated comment data.

8. A semantic deduplication device for comment data, characterized in that, include: The filtering module is used to filter the original comment data based on preset filtering rules to obtain duplicate comment data to be deduplicated. The preset filtering rules are used to filter comment data that meet the requirements of product relevance and authenticity. The determination module is used to determine candidate comment data from the comment data to be deduplicated based on a preset quantification rule, wherein the preset quantification rule is used to quantify and score the comment data to be deduplicated. The deduplication module is used to perform internal semantic deduplication and historical semantic deduplication on the candidate comment data to obtain deduplicated comment data.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 7 through the computer program.

Images