Non-response text duplicate removal method and device, equipment and medium
By using language models and iterative clustering algorithms to process the unresponsive text in the digital human scenario of auto insurance claims, the problem of low deduplication efficiency in traditional methods is solved, and efficient and accurate deduplication of unresponsive text is achieved, reducing manual intervention and costs.
Patent Information
- Application Number
- CN202510679508.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-23
- Publication Date
- 2025-10-17
AI Technical Summary
Traditional methods are unable to efficiently and accurately process the massive, diverse, and semantically complex unresponsive texts generated in the digital human scenario of auto insurance claims, resulting in an increase in redundant data and a reduction in the efficiency of knowledge base optimization and manual processing.
By obtaining the target prompt words and context information of the target consultation text, the language model is used to filter out the unanswered text, which is converted into word embedding vectors. The iterative clustering algorithm is used for clustering, and duplicate removal is performed based on the distance between the word embedding vector and the cluster center.
It improves the accuracy of screening unanswered texts, reduces redundant data, lowers manual annotation costs, and meets the needs of efficient and accurate data processing in modern production environments.
Smart Images
Figure CN120804314A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence, and is applied to online processing business scenarios such as finance and insurance, and particularly relates to a method and device for deduplicating non-answered texts, equipment and a medium. BACKGROUND
[0002] In the scenario of a car insurance claim digital person, customers describe accident details through multiple modalities such as text and voice, and submit thousands of natural language questions every day. However, due to insufficient coverage of the knowledge base, limited model capabilities, or diversity of expression, some questions cannot be effectively answered and are recorded as non-answered texts. These non-answered texts provide important data resources for knowledge base optimization, but traditional text processing techniques face many challenges when processing these non-answered texts.
[0003] First, there is a lot of semantically similar or repetitive content in non-answered texts, which not only occupies storage space but also increases the workload of labeling and classification in subsequent knowledge base expansion and optimization processes. Second, in a production environment, a large amount of non-answered texts are generated every day, and if they cannot be efficiently screened and deduplicated, redundant data will significantly reduce the efficiency of knowledge base optimization. In addition, the existence of repetitive and ineffective texts makes manual processing inefficient, slowing down the update and iteration speed of the knowledge base, further affecting the overall performance of the knowledge question and answer system. Traditional processing methods, such as keyword matching and rule filtering, as well as manual screening and labeling, all have significant defects. These methods cannot accurately capture the semantic similarity of texts, especially when dealing with a large number of non-answered texts with different expression forms but similar semantics, the efficiency and accuracy are limited. In addition, traditional methods lack automated and intelligent process design, making it difficult to meet the demand for efficient and accurate data processing in modern production environments. SUMMARY
[0004] The purpose of the embodiments of the present application is to propose a method and device for deduplicating non-answered texts, a computer device and a storage medium, to solve the problem that traditional text processing techniques cannot effectively process the large amount, diversity and complex semantics of non-answered texts generated in the scenario of a car insurance claim digital person, and cannot accurately deduplicate non-answered texts.
[0005] In a first aspect, a method for deduplicating non-answered texts is provided, which adopts the following technical solution:
[0006] When the trigger instruction of the target timing task is detected, the target prompt word and the context information corresponding to the target consultation text in the online dialogue corpus are acquired; based on the target prompt word and the context information, the target consultation text is filtered by using a preset language model, and the unresponded text corresponding to the target consultation text is extracted; the unresponded text is converted into a word embedding vector by using a preset text embedding model; based on the word embedding vector, the unresponded text is clustered by using a preset iterative clustering algorithm, and a clustering cluster of a plurality of clustering centers is determined; and the unresponded text is deduplicated based on the distance between each word embedding vector in the clustering cluster and the corresponding clustering center, to obtain the deduplicated unresponded text.
[0007] In a second aspect, a deduplication device for unresponded text is provided, which adopts the following technical solution:
[0008] The acquisition module is configured to acquire the target prompt word and the context information corresponding to the target consultation text in the online dialogue corpus when the trigger instruction of the target timing task is detected.
[0009] The filtering module is configured to filter the target consultation text by using a preset language model based on the target prompt word and the context information, and extract the unresponded text corresponding to the target consultation text.
[0010] The conversion module is configured to convert the unresponded text into a word embedding vector by using a preset text embedding model.
[0011] The clustering module is configured to cluster the unresponded text by using a preset iterative clustering algorithm based on the word embedding vector, and determine a clustering cluster of a plurality of clustering centers.
[0012] The deduplication module is configured to deduplicate the unresponded text based on the distance between each word embedding vector in the clustering cluster and the corresponding clustering center, to obtain the deduplicated unresponded text.
[0013] In a third aspect, a computer device is provided, which adopts the following technical solution:
[0014] When the trigger instruction of the target timing task is detected, the target prompt word and the context information corresponding to the target consultation text in the online dialogue corpus are acquired; based on the target prompt word and the context information, the target consultation text is filtered by using a preset language model, and the unresponded text corresponding to the target consultation text is extracted; the unresponded text is converted into a word embedding vector by using a preset text embedding model; based on the word embedding vector, the unresponded text is clustered by using a preset iterative clustering algorithm, and a clustering cluster of a plurality of clustering centers is determined; and the unresponded text is deduplicated based on the distance between each word embedding vector in the clustering cluster and the corresponding clustering center, to obtain the deduplicated unresponded text.
[0015] In a fourth aspect, a computer-readable storage medium is provided, and the following technical solutions are adopted:
[0016] When the trigger instruction of the target timing task is detected, the target prompt word and the context information corresponding to the target consultation text in the online dialogue corpus are acquired; based on the target prompt word and the context information, the target consultation text is filtered by using a preset language model, and the unresponded text corresponding to the target consultation text is extracted; the unresponded text is converted into a word embedding vector by using a preset text embedding model; and based on the word embedding vector, the unresponded text is clustered by using a preset iterative clustering algorithm to determine a clustering cluster of a plurality of clustering centers; and the unresponded text is de-duplicated based on the distance between each word embedding vector in the clustering cluster and the corresponding clustering center to obtain the de-duplicated unresponded text.
[0017] Compared with the prior art, the embodiments of the present application have the following beneficial effects: By using the target prompt word and the context information, the language model is used to filter the unresponded text of the target consultation text. The powerful semantic understanding capability of the language model can deeply analyze the text semantics and accurately identify the unresponded text. Compared with the traditional keyword matching and rule filtering, the effective unresponded text can be avoided to be missed due to semantic understanding deviation, and the filtering accuracy is greatly improved. The text embedding model is used to convert the unresponded text into a word embedding vector. This model can deeply mine the text semantics, map the texts with similar semantics but different expression forms to a similar vector space, and effectively solve the problem that the traditional statistical method is difficult to capture the deep semantic similarity. Based on the word embedding vector, the iterative clustering algorithm is used. The algorithm can dynamically adapt to the characteristics of the text data, accurately determine the clustering cluster of a plurality of clustering centers by continuously updating the clustering center, and de-duplicate the unresponded text according to the distance between the word embedding vector and the clustering center. Compared with the traditional method, the unresponded text with similar semantics or repetition can be accurately identified and removed, and the redundant data is greatly reduced. In addition, by constructing an automatic processing process, from the triggering of the timing task to the completion of the final de-duplication, no much manual intervention is needed throughout the process, the manual labeling cost is greatly reduced, efficient operation in a large-scale unresponded text environment is ensured, and the demand for efficient and accurate data processing in the modern production environment is met. BRIEF DESCRIPTION OF DRAWINGS
[0018] In order to more clearly illustrate the schemes in the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creating laborious work.
[0019] Figure 1 is an exemplary system architecture diagram to which the present application can be applied;
[0020] Figure 2A flow chart of one embodiment of the method for deduplicating non-interactive text according to the present application;
[0021] Figure 3 A structural schematic diagram of one embodiment of the deduplicating device for non-interactive text according to the present application;
[0022] Figure 4 A structural schematic diagram of one embodiment of the computer device according to the present application. DETAILED DESCRIPTION
[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application; the description herein and the claims of the application and the above description of the drawings herein is not intended to be complete descriptions of all features of the application; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. The terms "comprising," "having," "including," and "containing" used herein and in the following claims are each used open- ended. The terms "a," "an," and "the" used herein and in the following claims are each used in the sense that they can cover one or more instances of the element that they are used to describe. The terms "first," "second," and the like used herein and in the following claims are used to distinguish between two or more different elements.
[0024] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. As will be apparent to those of ordinary skill in the art, embodiments described herein can be combined with other embodiments.
[0025] In order to make the technical personnel in the art better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings.
[0026] As shown in Figure 1 The system architecture 100 can include a terminal device 101, a network 102, and a server 103, and the terminal device 101 can be a notebook computer 1011, a tablet computer 1012, or a mobile phone 1013. The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103. The network 102 can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.
[0027] A user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0028] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing, in addition to the notebook computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an electronic book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer and a desktop computer, etc.
[0029] The server 103 can be a server providing various services, for example, a background server providing support for a page displayed on the terminal device 101.
[0030] It should be noted that the method for deduplicating non-answered text provided by the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the deduplication device for non-answered text is generally arranged in a server / terminal device.
[0031] It should be understood that Figure 1 The number of terminal devices, networks and servers in
[0032] With reference to Figure 2 , a flow chart of one embodiment of the method for service recommendation according to the present application is shown. The method for deduplicating non-answered text comprises the following steps:
[0033] In step S201, when a trigger instruction of a target timing task is detected, target prompt words and context information corresponding to a target consultation text in an online dialogue corpus are acquired.
[0034] In the present embodiment, the electronic device (for example, the server / terminal device shown in Figure 1 ) on which the method for deduplicating non-answered text runs can acquire the target prompt words and the context information through a wired connection mode or a wireless connection mode. It should be noted that the wireless connection mode can include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other now known or future developed wireless connection modes.
[0035] The target timing task refers to a series of operation processes automatically triggered and executed at a specific time point or at fixed time intervals by the system. For example, in this embodiment, the unresponsive text processing task is started at the preset time. For example, the target timing task is automatically executed at 1 a.m. every day to start processing the unresponsive text of the previous day.
[0036] The trigger instruction is a specific signal or command used to start the execution of the target timing task. It is used to instruct the system to start executing the preset task. For example, when the system clock reaches the time point set by the target timing task, a trigger instruction is automatically generated to start the unresponsive text processing process.
[0037] The online dialogue corpus is a collection of text data generated by a large number of users in dialogue with the system, such as car insurance claim digital people. It comes from actual dialogue scenarios and represents a collection of real user communication content.
[0038] The target consultation text is the user consultation text selected as the processing object in the online dialogue corpus after the target timing task is triggered.
[0039] The target prompt word is a specific word or phrase used to guide the language model to accurately understand the intent and key points of the target consultation text.
[0040] The context information is additional information associated with the target consultation text that can help understand the semantic and background of the text. It comes from the dialogue process or text-related content, represents the environment and background of the text generation, and is used to help the language model more accurately determine whether the text is unresponsive text.
[0041] Step S202, based on the target prompt word and the context information, a preset language model is used to filter the target consultation text and extract the unresponsive text corresponding to the target consultation text.
[0042] The language model is an algorithm model trained on a large amount of text data and has the ability to understand, generate, and process natural language. It is used to filter unresponsive text based on target prompt words and context information. For example, the large language model LLM.
[0043] The unresponsive text is a user consultation text that cannot be effectively responded to in the car insurance claim digital person scenario due to insufficient knowledge base coverage, limited model capabilities, or diverse expression methods. For example, the text of a special claim scenario asked by the user is not replied by the system.
[0044] Step S203, a preset text embedding model is used to convert the unresponsive text into a word embedding vector.
[0045] The text embedding model is an algorithm model for converting text into a low-dimensional, dense vector representation. It is used to convert non-response text into word embedding vectors, such as the BGE-M3 model.
[0046] The word embedding vector is a low-dimensional, dense vector obtained by converting non-response text through a text embedding model. It represents the semantic features of non-response text in vector space and is used for subsequent clustering analysis. For example, a text describing car insurance claims is converted into a 128-dimensional word embedding vector.
[0047] Step S204: Based on the word embedding vector, a preset iterative clustering algorithm is used to cluster the non-response text to determine the clustering of multiple cluster centers.
[0048] The iterative clustering algorithm is an algorithm that continuously updates the cluster center to divide data points into different clusters. It is used to cluster non-response text based on word embedding vectors. For example, the K-Means clustering algorithm.
[0049] The multiple cluster centers are the center points representing the characteristics of each cluster determined during the execution of the iterative clustering algorithm. They are used to divide the clustering of non-response text. For example, three cluster centers are obtained through the iterative clustering algorithm, corresponding to three cluster clusters.
[0050] The clustering cluster is a collection of non-response text divided according to the similarity between word embedding vectors under the action of the iterative clustering algorithm. It is derived from the grouping result of the iterative clustering algorithm and represents a collection of non-response text with similar semantic features. For example, non-response texts with similar semantics are grouped into a cluster.
[0051] Step S205: Based on the distance between each word embedding vector in the cluster and the corresponding cluster center, the non-response text is de-duplicated to obtain the de-duplicated non-response text.
[0052] Each word embedding vector in the cluster refers to the word embedding vector corresponding to the non-response text belonging to the same cluster. For example, there are 50 word embedding vectors in a cluster.
[0053] The distance refers to the mathematical measure value between each word embedding vector in the cluster and the corresponding cluster center. It represents the similarity between the word embedding vector and the cluster center and is used to determine whether the non-response text is a duplicate. For example, the Euclidean distance is used to calculate the distance between vectors.
[0054] De-duplication refers to the operation of removing non-response texts with similar semantics or duplicates within a cluster based on the distance between each word embedding vector in the cluster and the corresponding cluster center. For example, duplicate car insurance claim consultation texts within the same cluster are removed.
[0055] The embodiment of the application can filter the non-responding text from the target consultation text by using the target prompt word and the context information and with the help of a language model. The powerful semantic understanding capability of the language model can deeply analyze the semantic of the text and accurately identify the non-responding text. Compared with the traditional keyword matching and rule filtering, the language model can avoid missing the effective non-responding text due to semantic understanding deviation and greatly improve the filtering accuracy. The text embedding model is used to convert the non-responding text into a word embedding vector. This model can deeply mine the semantic of the text and map the texts with similar semantics but different expression forms to a similar vector space, effectively solving the problem that the traditional statistical method is difficult to capture the deep semantic similarity. Based on the word embedding vector, an iterative clustering algorithm is used. The algorithm can dynamically adapt to the characteristics of the text data, accurately determine the clustering clusters of multiple clustering centers by continuously updating the clustering centers, and remove the redundant data. In addition, by constructing an automatic processing flow from the triggering of the timing task to the completion of the final deduplication, no more manual intervention is needed throughout the process, greatly reducing the cost of manual annotation, ensuring efficient operation in a large-scale non-responding text environment, and meeting the demand for efficient and accurate data processing in modern production environment.
[0056] In some optional implementation manners of the embodiment, step 201 of acquiring the target prompt word and the context information corresponding to the target consultation text in the online dialogue corpus comprises the following steps.
[0057] Based on the triggering instruction, the range of the text to be processed is determined from the online dialogue corpus; based on the range of the text to be processed, the target consultation text is determined from the online dialogue corpus; and based on the target consultation text, the target prompt word and the context information are constructed.
[0058] The range of the text to be processed refers to the boundary of the text set to be analyzed and processed subsequently, which is circled from the online dialogue corpus according to the target timing task triggering instruction. For example, when the target timing task is set to process the dialogue text between the user and the digital person for car insurance claim from 9:00 to 12:00 am, all the dialogue texts in this period constitute the range of the text to be processed.
[0059] In an example, in the digital human service system for vehicle insurance claims of a large insurance company, a large number of users have daily conversations with digital humans through online channels (such as APP, official website, etc.) to consult about claims related matters. Due to the limited coverage of the knowledge base, the diverse expression of users, and other reasons, part of the consultation cannot be effectively answered, resulting in a large amount of un-answered text. By triggering a timing task, the large language model is used to screen the text in the online conversation corpus, and the newly added un-answered text is accurately extracted every day. Specifically, when the trigger instruction of the target timing task is detected (for example, executed at 1 am every day), the system screens the conversation text between users and the digital human for vehicle insurance claims in the previous day from the online conversation corpus as the range of text to be processed according to the preset conditions in the instruction. In the range of text to be processed, the system further screens the text with complete question expression and closely related to the claims business as the target consultation text. For the target consultation text, the system constructs the target prompt words and context information through natural language processing technology. Taking "my car was rear-ended on the highway, the other party is fully liable, what materials do I need to prepare for claims" as an example, the target prompt words can be set as "vehicle insurance claims materials", "highway rear-end", "the other party is fully liable", etc. These prompt words can accurately summarize the core points of the text. The context information includes the vehicle model mentioned in the previous text, the insurance time, etc. to help the language model understand the user's intention more comprehensively.
[0060] The embodiment of the present application can determine the range of text to be processed through the trigger instruction, accurately frame the conversation text under specific time period or specific conditions, avoid resource waste of full data processing, and improve processing pertinence. Based on the range, the target consultation text is determined, which can filter irrelevant chatting or invalid information, focus on valuable consultation, and improve data quality. The construction of target prompt words and context information provides rich semantic guidance for subsequent analysis, makes the system understand the user's intention more deeply, and helps to efficiently screen un-answered text.
[0061] In some optional implementation manners, in step 202, based on the target prompt words and the context information, a preset language model is used to screen the target consultation text, and un-answered text corresponding to the target consultation text is extracted, specifically including the following steps:
[0062] The target prompt words, the context information, and the target consultation text are input into the preset language model, and the probability distribution information of the target consultation text is output; based on the probability distribution information, the un-answered text is extracted from the target consultation text.
[0063] The probability distribution information refers to a set of probability values about different categories obtained by the language model based on the target prompt word, the context information and the semantic understanding ability of the language model itself after inputting the target consultation text into the language model. For example, in the car insurance claim consultation scenario, the language model outputs that the probability of the target consultation text "My car is scratched in the rain, how does the insurance company handle the claim?" belonging to the "effective response" category is 70%, and the probability of belonging to the "no response" category is 30%, which constitutes the probability distribution information of the text.
[0064] In an example, when the user initiates the consultation, the system inputs the target prompt word, the context information and the target consultation text into the preset LLM language model. For example, the user consults "My new energy car is damaged in the rain, how does the insurance company handle the claim?", and the target prompt word is set as "new energy car", "rain", and "claim process". The context information includes the user's previous mention of the vehicle purchase time, insurance type, etc. After inputting these information into the language model, the model performs deep semantic analysis on the target consultation text based on the internal massive insurance knowledge data and algorithm logic, calculates the probability of the text belonging to different semantic categories (such as effective response, no response, etc.), and outputs the probability distribution information, such as the probability of the "effective response" category is 30%, and the probability of the "no response" category is 70%. According to the preset probability threshold (such as 50%), the system determines the probability distribution information of the target consultation text. Since the probability of the "no response" category in the above example is 70%, which is greater than the threshold of 50%, the system determines that the target consultation text is a no response text and extracts it.
[0065] The embodiments of the present application can deeply mine the semantic association of the text by integrating the target prompt word, the context information and the target consultation text into the language model. The language model is trained based on massive insurance knowledge, and the output probability distribution information accurately reflects the response possibility of the text, covering multiple potential semantic trends. Based on this information, the no response text can be effectively filtered to filter the interference text of the effective response.
[0066] In some optional implementation manners, in step 203, the no response text is converted into a word embedding vector by using a preset text embedding model, specifically including the following steps:
[0067] The no response text is preprocessed to obtain preprocessed text data, and the preprocessed text data is input into a preset text embedding model to obtain a word embedding vector.
[0068] The preprocessing refers to a series of standardization and normalization operation processes performed on the no response text. The preprocessing is used to eliminate noise, redundancy and format inconsistency in the no response text, so that the no response text is adapted to subsequent text embedding model processing.
[0069] The text data refers to data obtained by preprocessing the unresponsive text, which can be composed of language elements such as characters, words, sentences, etc.
[0070] In an example, data inclination, stop word removal, and text normalization are performed on all unresponsive texts to obtain preprocessed text data. The text data is input into a text embedding vector, such as a BGE-M3 text embedding model. The model is trained based on a large amount of insurance domain corpus and can accurately capture the semantics of the text. Taking the preprocessed text "I crashed my car into a tree and need to find the traffic police to determine responsibility for claim settlement" as an example, the model maps each word in the text to a high-dimensional vector space, considering the semantic association between words, context information, etc., to generate corresponding word embedding vectors.
[0071] The embodiments of the present application can eliminate noise interference such as punctuation disorder, colloquial redundancy, and format inconsistency in the original text by preprocessing the unresponsive text. The preprocessed text data is input into the text embedding model, and the model can accurately capture the deep semantic association of the text and convert the text into word embedding vectors with rich semantic representation. This vector not only reduces the data dimension but also strengthens the semantic distinction, facilitating subsequent similar text clustering, knowledge mining, etc.
[0072] In some optional implementations, step 204, based on the word embedding vector, a preset iterative clustering algorithm is used to cluster the unresponsive text to determine a plurality of clustering centers of the clustering cluster, specifically including the following steps:
[0073] Based on the preset iterative clustering algorithm, a plurality of target word embedding vectors are determined from the word embedding vectors as clustering centers; the target distance between each word embedding vector and each clustering center is calculated; based on the target distance, the word embedding vector is assigned to the corresponding clustering center to obtain a plurality of clustering clusters; the mean value of the word embedding vectors in each clustering cluster is calculated; the clustering center is updated based on the mean value, and the plurality of clustering clusters are updated based on the updated clustering center; the step of assigning the word embedding vector to the clustering center and updating the clustering center is returned until the obtained clustering center satisfies the preset termination condition.
[0074] The plurality of target word embedding vectors refers to a number of representative vectors selected from the word embedding vectors based on a specific rule. They are used as initial clustering centers to assist subsequent clustering and grouping of all word embedding vectors. For example, in the processing of unresponsive text for car insurance claim settlement, 10 word embedding vectors are selected as initial target word embedding vectors by random sampling or based on semantic distribution, which will guide the classification of other vectors.
[0075] The clustering center is a word embedding vector representing the core semantic features of a clustering cluster during the execution of the iterative clustering algorithm.
[0076] wherein the target distance refers to a quantitative indicator for measuring the semantic similarity degree between the word embedding vector and the cluster center. It represents the relative position relationship of the two vectors in the semantic space, and is used to determine the cluster to which the word embedding vector belongs, thereby assisting the semantic clustering of the text.
[0077] wherein the plurality of cluster clusters refers to a plurality of sets formed by dividing all word embedding vectors according to the semantic similarity degree based on the iterative clustering algorithm. It represents a set of non-response texts of different semantic categories, and is used to distinguish non-response texts of different themes or semantics.
[0078] wherein the mean value refers to the average value of the numerical value of each dimension of all word embedding vectors in the cluster cluster in the vector space.
[0079] wherein the termination condition refers to a criterion for determining whether the iterative clustering algorithm is stopped. For example, in the clustering of non-response texts of car insurance claims, the termination condition is set as the cluster center no longer changing or reaching a preset number of iterations (such as 50 times). When the variation amplitude of each dimension of the cluster center vector is less than 0.001 after two consecutive iterations, it is considered that the cluster center no longer changes, and the algorithm stops iterating. Or when the number of iterations reaches 50 times, regardless of whether the cluster center changes, the algorithm stops running.
[0080] In an example, taking the K-Means algorithm as an example, 5 representative vectors are randomly selected from the set of word embedding vectors converted from the non-response texts of car insurance claims as initial cluster centers. For example, vectors related to “car insurance claim process”, “claim material requirements”, “out-of-town accident handling”, “exemption clause consultation”, “premium adjustment problem” and other different semantic directions are selected. The cosine similarity between each word embedding vector and each cluster center is calculated as the target distance. If the cosine similarity between a word embedding vector and the “car insurance claim process” cluster center is 0.85, and the cosine similarity between the word embedding vector and the “claim material requirements” cluster center is 0.7, then the vector is assigned to the “car insurance claim process” cluster. The mean value of all word embedding vectors in each dimension of each cluster is calculated to obtain a new cluster center vector. For example, there are 20 vectors in the “car insurance claim process” cluster, and the average of their numerical values in each dimension is calculated to generate a new cluster center vector. Based on the new cluster center, the word embedding vector is re-assigned to the corresponding cluster to obtain an updated cluster. Repeat the above assignment and update steps until the cluster center no longer changes or reaches a preset number of iterations (such as 30 times). When the variation amplitude of each dimension of each cluster center vector is less than 0.001 after two consecutive iterations, it is determined that the cluster center is stable, and the algorithm is terminated.
[0081] The embodiments of the present application anchor the core semantic points for subsequent precise clustering by starting from determining a plurality of target word embedding vectors as initial clustering centers. By calculating the target distance of each word embedding vector from the clustering center, the semantic similarity is quantified, ensuring that the vector is assigned to the most suitable clustering cluster, so that the clustering result is more in line with the actual text semantics. The clustering center is updated by the mean value, and the clustering core is dynamically adjusted, so that the clustering center can accurately reflect the semantic features of the vectors in the cluster, enhancing the stability and accuracy of clustering. Repeat the assignment and update steps until the termination condition is met. This process is like a constantly polished sieve, gradually optimizing the clustering effect. The final clustering cluster can clearly distinguish different semantics of non-response text.
[0082] In some optional implementations, the step of "calculating the mean value of the word embedding vectors in each clustering cluster" specifically includes the following steps:
[0083] Obtain the number of word embedding vectors in each clustering cluster; sum all the word embedding vectors in each clustering cluster to obtain the sum of the vectors in the cluster; and average the number of the sum of the vectors in the cluster to obtain the mean value of the word embedding vectors in each clustering cluster.
[0084] The sum of the vectors in the cluster refers to the vector result obtained by element-by-element addition of the numerical values of all word embedding vectors in the same clustering cluster in each dimension of the vector space. For example, in a certain cluster formed by clustering non-response text of car insurance claims, there are 3 word embedding vectors, [0.2, 0.5, 0.3], [0.1, 0.3, 0.6], and [0.4, 0.2, 0.1]. Adding the corresponding dimension values, the sum of the vectors in the cluster is [0.2+0.1+0.4, 0.5+0.3+0.2, 0.3+0.6+0.1] = [0.7, 1.0, 1.0].
[0085] The average refers to the calculation process of dividing the sum of the vectors in the cluster in the cluster by the number of word embedding vectors in the cluster, thereby obtaining a vector that can represent the average semantic features of the cluster.
[0086] In an example, based on the iterative clustering algorithm, the initial clustering center is determined from the word embedding vectors converted from the car insurance claim non-response text, and after multiple iterations of assignment and update, a plurality of stable clustering clusters are obtained, such as "claim material class", "claim process class", "exemption clause class", etc. For each clustering cluster, the number of word embedding vectors contained therein is counted. For example, there are 15 word embedding vectors in the "claim material class" clustering cluster, and these vectors correspond to different user consultation texts about the materials required for car insurance claim. The element-by-element addition is performed on all word embedding vectors in each clustering cluster in each dimension of the vector space. Taking the "claim material class" as an example, if there are vectors [0.2, 0.5, 0.3], [0.1, 0.3, 0.6]… in the cluster, the corresponding dimension values are added to obtain the vector sum in the cluster. The vector sum in the cluster is divided by the number of word embedding vectors in the cluster. For example, the vector sum in the "claim material class" cluster is [3.0, 7.5, 4.5], the number of vectors is 15, and the mean value is [3.0 / 15, 7.5 / 15, 4.5 / 15] = [0.2, 0.5, 0.3]. The clustering center is updated based on the calculated mean value, and the word embedding vectors are re-assigned to the corresponding cluster according to the new clustering center, and the mean value is calculated again. Repeat this process until the clustering center meets the termination condition.
[0087] The embodiments of the present application can provide basic data support for subsequent calculation by counting the number of vectors in the cluster, and ensure the accuracy of the operation. Summing and averaging the vectors in the cluster can accurately extract the semantic core features of each clustering cluster, making the clustering center more consistent with the actual semantics of the text in the cluster. After updating the clustering center based on the mean value, the subsequent vector assignment is more reasonable, greatly improving the clustering quality and making the non-response texts of different semantic categories more clearly distinguishable, providing a solid guarantee for subsequent efficient deduplication.
[0088] In some optional implementation manners, in step S206, the non-response text is deduplicated based on the distance between each word embedding vector in the clustering cluster and the corresponding clustering center, to obtain the deduplicated non-response text, and specifically includes the following steps:
[0089] The distance between each word embedding vector in each clustering cluster and the corresponding clustering center is calculated. The word embedding vectors with a distance greater than a preset distance threshold are removed, and the deduplicated non-response text is obtained.
[0090] In an example, the Euclidean distance between each word embedding vector in each cluster and its corresponding clustering center can be calculated. The distance threshold d is set to 0.5. For example, the distance between vector A in a certain cluster and the clustering center is 0.6 (> 0.5), and the corresponding non-response text "import drug reimbursement policy?" is removed. The text "million medical insurance import drug coverage" with a distance of 0.3 (< 0.5) is retained.
[0091] The embodiment of the application can objectively measure the dispersion degree of each text in the semantic space based on the distance calculation of the word embedding vector and the cluster center. The quantitative index effectively overcomes the limitation of the traditional deduplication method which only relies on the surface similarity of the text. The abnormal vector whose distance exceeds the preset threshold is removed in a targeted manner, avoiding the effective deletion of effective texts with semantic relevance but different expressions caused by hard matching. The unresponsive text set processed in this way eliminates redundant information while maintaining semantic diversity.
[0092] It should be emphasized that, in order to further ensure the privacy and security of the above-mentioned target prompt word, context information, unresponsive text, word embedding vector, and reprocessed unresponsive text, the above-mentioned target prompt word, context information, unresponsive text, word embedding vector, and reprocessed unresponsive text can also be stored in a node of a blockchain.
[0093] The blockchain referred to in the present application is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm and other computer technologies. Blockchain, in essence, is a decentralized database, a series of data blocks associated using cryptographic methods, each containing a batch of network transaction information for verifying the validity (anti-fake) of the information and generating the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.
[0094] The embodiment of the application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0095] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.
[0096] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware through computer readable instructions, and the computer readable instructions can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiment methods. Among them, the storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0097] It should be understood that although each step in the flowchart of the accompanying drawings is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other orders. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be alternately executed with other steps or sub-steps or stages of other steps.
[0098] Further referring to Figure 3 , as an implementation of the method shown in Figure 2 , the present application provides an embodiment of a device for deduplicating unresponded text, which corresponds to the method embodiment shown in Figure 2 , and the device can be specifically applied to various electronic devices.
[0099] As shown in Figure 4 , the device for deduplicating unresponded text 400 of the present embodiment comprises an acquisition module 401, a screening module 402, a conversion module 403, a clustering module 404, and a deduplication module 405. Among them:
[0100] The acquisition module 401 is configured to acquire the target prompt word and the context information corresponding to the target consultation text in the online dialogue corpus when detecting the trigger instruction of the target timing task;
[0101] The screening module 402 is configured to screen the target consultation text based on the target prompt word and the context information, and extract the unresponded text corresponding to the target consultation text by using a preset language model;
[0102] The conversion module 403 is configured to convert the unresponded text into a word embedding vector by using a preset text embedding model;
[0103] The clustering module 404 is configured to cluster the unresponsive text based on the word embedding vectors, adopt a preset iterative clustering algorithm, and determine a clustering cluster of a plurality of clustering centers.
[0104] The deduplication module 405 is configured to deduplicate the unresponsive text based on a distance between each word embedding vector in the clustering cluster and the corresponding clustering center, and obtain deduplicated unresponsive text.
[0105] In this embodiment, the target prompt word and the context information can be used to filter the unresponsive text from the target consultation text by means of the language model. The language model has strong semantic understanding capability and can deeply analyze the semantic of the text and accurately identify the unresponsive text. Compared with the traditional keyword matching and rule filtering, the language model can avoid missing effective unresponsive text due to semantic understanding deviation, and greatly improve the filtering accuracy. The text embedding model is used to convert the unresponsive text into word embedding vectors. This model can deeply mine the semantic of the text, map the texts with similar semantic but different expression forms to a similar vector space, and effectively solve the problem that the traditional statistical method is difficult to capture the deep semantic similarity. Based on the word embedding vectors, the iterative clustering algorithm is used. The algorithm can dynamically adapt to the characteristics of the text data, accurately determine the clustering cluster of a plurality of clustering centers by continuously updating the clustering center, and then deduplicate the unresponsive text according to the distance between the word embedding vector and the clustering center. Compared with the traditional method, the language model can accurately identify and remove the unresponsive text with similar or repeated semantic, and greatly reduce the redundant data. In addition, by constructing an automatic processing process, the process from triggering the timing task to completing the deduplication does not require much manual intervention, greatly reduces the cost of manual annotation, ensures efficient operation in a large-scale unresponsive text environment, and meets the demand for efficient and accurate data processing in modern production environment.
[0106] In an embodiment, the acquisition module 401 comprises:
[0107] The range determination sub-module is configured to determine a text range to be processed from the online dialogue corpus based on the triggering instruction.
[0108] The text determination sub-module is configured to determine target consultation text from the online dialogue corpus based on the text range to be processed.
[0109] The construction sub-module is configured to construct a target prompt word and context information based on the target consultation text.
[0110] The embodiments of the present application can determine the text range to be processed by the triggering instruction, accurately frame the dialogue text under specific period or specific conditions, avoid resource waste of full data processing, and improve the processing pertinence. Based on the range, the target consultation text can be determined to filter irrelevant chitchat or invalid information, focus on valuable consultation, and improve the data quality. The construction of the target prompt word and the context information provides rich semantic guidance for subsequent analysis, so that the system can better understand the user's intention, and help to efficiently filter the unresponsive text.
[0111] In an embodiment, the screening module 402 comprises:
[0112] a first input sub-module configured to input the target prompt word, the context information and the target consultation text into a preset language model, and output probability distribution information of the target consultation text;
[0113] a determination sub-module configured to extract non-responding text from the target consultation text based on the probability distribution information.
[0114] By integrating the target prompt word, the context information and the target consultation text into the language model, the embodiment of the present application can deeply mine the semantic association of the text. The language model is trained based on a large amount of insurance knowledge, and the output probability distribution information accurately reflects the possibility of text response, covering various potential semantic trends. Based on this information, the non-responding text can be extracted, which can effectively filter the interfering text that can be effectively responded.
[0115] In an embodiment, the conversion module 403 comprises:
[0116] a preprocessing sub-module configured to preprocess the non-responding text to obtain preprocessed text data;
[0117] a second input sub-module configured to input the preprocessed text data into a preset text embedding model to obtain a word embedding vector.
[0118] By preprocessing the non-responding text, the embodiment of the present application can eliminate noise interference such as punctuation disorder, colloquial redundancy and format inconsistency in the original text. After the preprocessed text data is input into the text embedding model, the model can accurately capture the deep semantic association of the text and convert the text into a word embedding vector with rich semantic representation. This vector not only reduces the data dimension, but also strengthens the semantic distinction, which is convenient for subsequent similar text clustering, knowledge mining and other operations.
[0119] In an embodiment, the clustering module 404 comprises:
[0120] a center determination sub-module configured to determine a plurality of target word embedding vectors as clustering centers from the word embedding vectors based on a preset iterative clustering algorithm;
[0121] a first calculation sub-module configured to calculate a target distance between each word embedding vector and each clustering center;
[0122] an assignment sub-module configured to assign the word embedding vector to the corresponding clustering center based on the target distance to obtain a plurality of clustering clusters;
[0123] a second calculation sub-module configured to calculate the mean of the word embedding vectors in each clustering cluster;
[0124] The updating sub-module is configured to update the cluster centers based on the mean values, and update the plurality of cluster clusters based on the updated cluster centers.
[0125] The performing sub-module is configured to return to performing the steps of assigning the word embedding vectors to the cluster centers and updating the cluster centers until the obtained cluster centers satisfy a preset termination condition.
[0126] The embodiments of the present application anchor the core semantic points for subsequent accurate clustering by determining a plurality of target word embedding vectors as initial cluster centers. By calculating the target distance between each word embedding vector and the cluster center, the semantic similarity is quantified, ensuring that the vector is assigned to the most suitable cluster, so that the clustering result is more in line with the actual text semantics. The cluster centers are updated by the mean value, and the cluster core is dynamically adjusted, so that the cluster centers can accurately reflect the semantic features of the vectors in the cluster, and the clustering stability and accuracy are enhanced. The steps of assignment and updating are repeated until the termination condition is met. This process is like a constantly polished sieve, gradually optimizing the clustering effect. The final obtained cluster can clearly distinguish different semantic non-response texts.
[0127] In an embodiment, the mean value calculation sub-module is further configured to obtain the number of word embedding vectors in each cluster; sum all the word embedding vectors in each cluster to obtain the sum of the vectors in the cluster; and average the number of the sum of the vectors in the cluster to obtain the mean value of the word embedding vectors in each cluster.
[0128] The embodiments of the present application can provide basic data support for subsequent calculation by counting the number of vectors in the cluster, ensuring the accuracy of the operation. Summing and averaging the vectors in the cluster can accurately extract the semantic core features of each cluster, making the cluster center more in line with the actual semantics of the text in the cluster. Based on this, the subsequent vector assignment is more reasonable after updating the cluster center based on the mean value, greatly improving the clustering quality, and making the non-response texts of different semantic categories more clearly distinguishable, providing a solid guarantee for subsequent efficient deduplication.
[0129] In an embodiment, the deduplication module 405 comprises:
[0130] The third calculation sub-module is configured to calculate the distance between each word embedding vector in each cluster and the corresponding cluster center.
[0131] The removal sub-module is configured to remove the word embedding vectors whose distance is greater than a preset distance threshold, and the corresponding non-response text, to obtain the deduplicated non-response text.
[0132] The embodiment of the application can objectively measure the dispersion degree of each text in the semantic space based on the distance calculation of the word embedding vector and the cluster center. The quantitative index effectively overcomes the limitation of the traditional deduplication method which only relies on the surface similarity of the text. The abnormal vector whose distance exceeds the preset threshold is removed in a targeted manner, avoiding the effective deletion of the valid text which is semantically related but has different expressions due to hard matching. The unresponsive text set processed in this way eliminates the redundant information while maintaining semantic diversity.
[0133] To solve the above technical problems, the embodiment of the application further provides a computer device. For details, please refer to Figure 4 , Figure 4 The basic structure block diagram of the computer device of the embodiment is shown in the figure.
[0134] The computer device 4 includes a memory 61, a processor 62, and a network interface 63 which are connected to each other through a system bus. It should be noted that only the computer device 6 with the memory 61, the processor 62, and the network interface 63 is shown in the figure, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented. Among them, the computer device herein is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.
[0135] The computer device can be a desktop computer, a notebook computer, a palm computer, and a cloud server, etc. The computer device can interact with the user through a keyboard, a mouse, a remote controller, a touchpad, a voice control device, etc.
[0136] The memory 61 includes at least one type of readable storage medium, including a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or a memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 6. Of course, the memory 61 can include both an internal storage unit and an external storage device of the computer device 6. In the present embodiment, the memory 61 is generally used to store an operating system and various application software installed on the computer device 6, such as computer readable instructions of the method for deduplicating unresponded text, etc. In addition, the memory 61 can also be used to temporarily store various data that have been output or will be output.
[0137] The processor 62 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In the present embodiment, the processor 62 is used to run computer readable instructions or process data stored in the memory 61, such as computer readable instructions of the method for deduplicating unresponded text.
[0138] The network interface 63 can include a wireless network interface or a wired network interface, and is generally used to establish a communication connection between the computer device 6 and other electronic devices.
[0139] The embodiment of the application can filter the non-response text of the target consultation text by using the target prompt word and the context information with the help of the language model. The powerful semantic understanding ability of the language model can deeply analyze the text semantics and accurately identify the non-response text. Compared with the traditional keyword matching and rule filtering, the powerful semantic understanding ability of the language model can avoid missing effective non-response text due to semantic understanding deviation and greatly improve the filtering accuracy. The text embedding model is used to convert the non-response text into a word embedding vector. This model can deeply mine the text semantics and map the texts with similar semantics but different expression forms to a similar vector space, effectively solving the problem that the traditional statistical method is difficult to capture the deep semantic similarity. Based on the word embedding vector, an iterative clustering algorithm is used. The algorithm can dynamically adapt to the characteristics of the text data, accurately determine the clustering clusters of multiple clustering centers by continuously iterating and updating the clustering centers. Then, the word embedding vector and the clustering center distance are used for deduplication. Compared with the traditional method, the word embedding vector and the clustering center distance can accurately identify and remove the non-response texts with similar or repeated semantics, greatly reducing the redundant data. In addition, by constructing an automatic processing process from the triggering of a timing task to the final completion of deduplication, no more manual intervention is needed throughout the process, greatly reducing the cost of manual annotation, ensuring efficient operation in a large-scale non-response text environment, and meeting the demand for efficient and accurate data processing in the modern production environment.
[0140] The application also provides another embodiment, that is, a computer readable storage medium storing computer readable instructions executable by at least one processor to cause the at least one processor to perform the steps of the non-response text deduplication method as described above.
[0141] The embodiment of the application can filter the non-response text of the target consultation text by using the target prompt word and the context information with the help of the language model. The powerful semantic understanding ability of the language model can deeply analyze the text semantics and accurately identify the non-response text. Compared with the traditional keyword matching and rule filtering, the powerful semantic understanding ability of the language model can avoid missing effective non-response text due to semantic understanding deviation and greatly improve the filtering accuracy. The text embedding model is used to convert the non-response text into a word embedding vector. This model can deeply mine the text semantics and map the texts with similar semantics but different expression forms to a similar vector space, effectively solving the problem that the traditional statistical method is difficult to capture the deep semantic similarity. Based on the word embedding vector, an iterative clustering algorithm is used. The algorithm can dynamically adapt to the characteristics of the text data, accurately determine the clustering clusters of multiple clustering centers by continuously iterating and updating the clustering centers. Then, the word embedding vector and the clustering center distance are used for deduplication. Compared with the traditional method, the word embedding vector and the clustering center distance can accurately identify and remove the non-response texts with similar or repeated semantics, greatly reducing the redundant data. In addition, by constructing an automatic processing process from the triggering of a timing task to the final completion of deduplication, no more manual intervention is needed throughout the process, greatly reducing the cost of manual annotation, ensuring efficient operation in a large-scale non-response text environment, and meeting the demand for efficient and accurate data processing in the modern production environment.
[0142] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned example method can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product in essence or in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disc, optical disc), and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the method of each embodiment of the present application.
[0143] Obviously, the above-described embodiments are only some of the embodiments of the present application, not all the embodiments, and the preferred embodiments of the present application are given in the drawings, but do not limit the patent scope of the present application. The present application can be implemented in many different forms, and conversely, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing specific embodiments, or equivalently replace some technical features. Any equivalent structure made by using the content of the specification and drawings, directly or indirectly applied to other related technical fields, is also within the scope of the patent protection of the present application.
[0144] The non-company software tools or components appearing in the embodiments of the present application are only examples for introduction, not representing actual use.
Claims
1. A method for removing duplicates of unanswered texts, characterized in that: The steps include: When a trigger instruction of a target timed task is detected, the target prompt word and context information corresponding to the target consultation text in the online dialogue corpus are obtained; Based on the target prompt word and the context information, the target consultation text is screened using a preset language model to extract the unanswered text corresponding to the target consultation text; Using a preset text embedding model, converting the unresponsive text into a word embedding vector; Based on the word embedding vector, a preset iterative clustering algorithm is used to cluster the unresponsive texts to determine cluster clusters with multiple cluster centers; Based on the distance between each word embedding vector in the cluster and the corresponding cluster center, the unresponsive text is deduplicated to obtain a deduplicated unresponsive text.
2. The method according to claim 1, characterized in that The step of obtaining target prompt words and context information corresponding to the target consultation text in the online dialogue corpus specifically includes: Based on the trigger instruction, determining a text range to be processed from an online dialogue corpus; Based on the scope of the text to be processed, determining a target consultation text from the online dialogue corpus; Based on the target consultation text, target prompt words and context information are constructed.
3. The method according to claim 1, characterized in that The step of screening the target consultation text using a preset language model based on the target prompt word and the context information and extracting the unanswered text corresponding to the target consultation text specifically includes: Inputting the target prompt word, the context information, and the target consultation text into a preset language model, and outputting probability distribution information of the target consultation text; Based on the probability distribution information, unanswered text is extracted from the target consultation text.
4. The method according to claim 1, wherein The step of converting the unresponsive text into a word embedding vector using a preset text embedding model specifically includes: Preprocessing the unanswered text to obtain preprocessed text data; The preprocessed text data is input into a preset text embedding model to obtain a word embedding vector.
5. The method according to claim 1, wherein The step of clustering the unresponsive texts based on the word embedding vectors using a preset iterative clustering algorithm to determine clusters of multiple cluster centers specifically includes: Based on a preset iterative clustering algorithm, determining a plurality of target word embedding vectors from the word embedding vectors as cluster centers; Calculate the target distance between each word embedding vector and each cluster center; Based on the target distance, the word embedding vector is assigned to the corresponding cluster center to obtain multiple clusters; Calculate the mean of the word embedding vectors within each cluster; updating the cluster center based on the mean, and updating the plurality of clusters based on the updated cluster center; Return to the step of assigning word embedding vectors to cluster centers and updating the cluster centers until the obtained cluster centers meet the preset termination conditions.
6. The method according to claim 5, characterized in that The step of calculating the mean of the word embedding vectors within each cluster specifically includes: Get the number of word embedding vectors in each cluster; Sum all word embedding vectors within each cluster to obtain the sum of the vectors within the cluster; The sum of the vectors within the cluster is averaged to obtain the mean of the word embedding vectors within each cluster.
7. The method according to claim 1, characterized in that The step of removing duplicates from the unresponsive text based on the distance between each word embedding vector in the cluster and the corresponding cluster center to obtain the removed duplicate unresponsive text specifically includes: Calculate the distance between each word embedding vector and the corresponding cluster center in each cluster; The word embedding vectors whose distance is greater than a preset distance threshold and the corresponding unanswered texts are removed to obtain the deduplicated unanswered texts.
8. A device for removing duplicates of unanswered texts, characterized in that: include: An acquisition module is used to acquire target prompt words and context information corresponding to target consultation text in the online dialogue corpus when a trigger instruction of a target timed task is detected; A screening module, configured to screen the target consultation text using a preset language model based on the target prompt word and the context information, and extract the unanswered text corresponding to the target consultation text; A conversion module, configured to convert the unresponsive text into a word embedding vector using a preset text embedding model; A clustering module, configured to cluster the unresponsive texts based on the word embedding vectors using a preset iterative clustering algorithm to determine clusters of multiple cluster centers; The deduplication module is used to deduplicate the unresponsive text based on the distance between each word embedding vector in the cluster and the corresponding cluster center to obtain the deduplicated unresponsive text.
9. A computer device, characterized in that: It includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of the method for deduplicating unresponsive texts as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the method for deduplicating unresponsive texts as described in any one of claims 1 to 7.