Data processing method and device, equipment and storage medium
By classifying and summarizing data using spectral clustering and Textrank algorithms, the problem of reducing useless data in massive datasets is solved, thereby reducing the amount of data and improving its applicability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA TELECOM DIGITAL INTELLIGENCE TECH CO LTD
- Filing Date
- 2022-09-01
- Publication Date
- 2026-06-05
AI Technical Summary
How to streamline useless data from massive datasets and improve the applicability and efficiency of data analysis.
The data is clustered into preset categories using spectral clustering algorithm, key data and general summaries are extracted using Textrank algorithm, and specific summaries are generated through preset model, eliminating invalid data and retaining key elements.
While retaining key data elements, reduce the amount of data to improve its applicability and analytical efficiency.
Smart Images
Figure CN115329087B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and in particular to a data processing method, apparatus, device and storage medium. Background Technology
[0002] As society enters the information age, the volume of data is gradually increasing. Users need to process a large amount of data every day, but much of this data is useless. How to simplify this massive amount of data is a pressing problem that needs to be solved. Summary of the Invention
[0003] This disclosure provides a data processing method, apparatus, device, and storage medium that overcomes, to some extent, the current problem of large data volumes that are difficult to simplify.
[0004] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part by practice of this disclosure.
[0005] According to one aspect of this disclosure, a data processing method is provided, comprising:
[0006] Acquire data, wherein the data is represented as text information;
[0007] The data is clustered into preset categories based on a spectral clustering algorithm;
[0008] In each category, a predetermined number of key data points and key data extracted according to the Textrank algorithm are used as a summary.
[0009] Extract specific summaries for each category based on a preset model;
[0010] The general summary and specific summary of the current category are determined as the target summary of the current category.
[0011] In one embodiment of this disclosure, after determining the general summary and specific summary of the current category as the target summary of the current category, the method further includes:
[0012] The target digest is sent to the user equipment so that the user can correct the target digest.
[0013] In one embodiment of this disclosure, after acquiring data, including data represented by text information, and before clustering the data into preset categories according to a spectral clustering algorithm, the method further includes:
[0014] The acquired data is filtered according to data standards to obtain the filtered data;
[0015] The filtered data is segmented into words according to the word segmentation criteria and algorithm to obtain segmented data.
[0016] Based on the synonym criteria, the synonyms in the segmented data are replaced with standard words to obtain the replaced data.
[0017] In one embodiment of this disclosure, before clustering the data into preset categories according to a spectral clustering algorithm, the method further includes:
[0018] The data is clustered multiple times according to different numbers of categories to obtain the silhouette coefficients corresponding to each number of categories.
[0019] The number of categories corresponding to the largest silhouette coefficient is set to a preset number.
[0020] In one embodiment of this disclosure, a predetermined number of key data points and key data extracted according to the Textrank algorithm are used as a summary in each category, including:
[0021] Based on the TF-IDF algorithm, a predetermined number of key data points are extracted from the data in each category;
[0022] Extract key data from each category based on the TextRank algorithm;
[0023] A summary is obtained by integrating key data with important data.
[0024] In one embodiment of this disclosure, before extracting the specific summary corresponding to each category according to a preset model in each category, the method further includes:
[0025] The model is trained based on historical data, labeled text, and historical target text, and a preset model is obtained after the training stops.
[0026] In one embodiment of this disclosure,
[0027] Within each category, extract the specific summary corresponding to each category based on the preset model, including:
[0028] For each category, a summary corresponding to each category is extracted based on a supervised generative model;
[0029] The summaries corresponding to each category are merged to obtain the summary text for each category.
[0030] Input the summary text corresponding to each category into the summary extraction model to obtain the specific summary corresponding to each category.
[0031] According to another aspect of this disclosure, a data processing apparatus is provided, the apparatus comprising:
[0032] The acquisition module is used to acquire data, wherein the data is represented as text information;
[0033] The first clustering module is used to cluster the data into preset categories according to the spectral clustering algorithm;
[0034] The first extraction module is used to extract a preset number of key data points and key data extracted according to the Textrank algorithm as a summary in each category;
[0035] The second extraction module is used to extract the specific summary corresponding to each category based on a preset model.
[0036] The first determining module is used to determine the general summary and specific summary of the current category as the target summary of the current category.
[0037] In one embodiment of this disclosure, the data processing apparatus further includes:
[0038] The sending module, after determining the general summary and specific summary of the current category as the target summary of the current category, sends the target summary to the user equipment so that the user can correct the target summary.
[0039] In one embodiment of this disclosure, the data processing apparatus further includes:
[0040] The filtering module, after acquiring data, including data represented by text information, and before clustering the data into preset categories according to the spectral clustering algorithm, is used to filter the acquired data according to data standards to obtain filtered data;
[0041] The word segmentation module is used to segment the filtered data into words according to the word segmentation criteria and algorithms, and obtain the segmented data.
[0042] The replacement module is used to replace synonyms in the segmented data with standard words based on the synonym criteria, so as to obtain the replaced data.
[0043] In one embodiment of this disclosure, the data processing apparatus further includes:
[0044] The second clustering module is used to cluster the data into preset categories according to the spectral clustering algorithm, and then cluster the data multiple times according to different numbers of categories to obtain the contour coefficients after clustering for each number of categories.
[0045] The second determining module is used to determine the number of categories corresponding to the largest contour coefficient as a preset number.
[0046] In one embodiment of this disclosure, the extraction module further includes:
[0047] The first extraction unit is used to extract a preset number of key data points from each category of data according to the TF-IDF algorithm.
[0048] The second extraction unit is used to extract key data for each category of data based on the Textrank algorithm;
[0049] The first fusion unit is used to merge key data with important data to obtain a summary.
[0050] In one embodiment of this disclosure, the data processing apparatus further includes:
[0051] The training module, before extracting the specific summary corresponding to each category according to the preset model, is used to train the model based on historical data, labeled text, and historical target text. The preset model is obtained after the training stopping condition is met.
[0052] In one embodiment of this disclosure, the second extraction module includes:
[0053] The third extraction unit is used to extract the summary corresponding to each category based on the supervised generation model.
[0054] The second fusion unit is used to fuse the summaries corresponding to each category to obtain the summary text corresponding to each category.
[0055] The extraction unit inputs the summary text corresponding to each category into the summary extraction model to obtain the specific summary corresponding to each category.
[0056] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-described data processing method by executing the executable instructions.
[0057] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described data processing method.
[0058] The data processing method provided in the embodiments of this disclosure acquires data, then clusters the data into preset categories using a spectral clustering algorithm. Within each category, a preset number of key data points and key data extracted using the Textrank algorithm are used as a general summary. Within each category, a specific summary corresponding to each category is extracted according to a preset model. The general summary and specific summary of the current category are then determined as the target summary for that category. Because the data is extracted, invalid data is eliminated. Furthermore, because the specific and general summaries of the data are acquired and determined as the target summary, the data volume is small while retaining its key elements, making the data easier to utilize and improving its applicability.
[0059] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0060] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.
[0061] Figure 1 This diagram illustrates a data processing method according to an embodiment of the present disclosure.
[0062] Figure 2 This diagram illustrates a data processing apparatus according to an embodiment of the present disclosure.
[0063] Figure 3 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation
[0064] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0065] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.
[0066] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.
[0067] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.
[0068] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".
[0069] As telecom operators' services become increasingly diversified and refined, new products and marketing methods are constantly being introduced. However, if these new products, services, and marketing approaches fail to gain user acceptance and satisfaction, they may lead to a series of complaints, even escalating to higher levels. On the other hand, while these complaints put pressure on operators' operations, they also provide directions and suggestions for improvement, reminding marketing personnel which products and methods are still unsatisfactory and which services are currently under intense scrutiny. However, with a large number of complaints each month, marketing personnel cannot review each one individually. To identify the main issues raised in the complaints, they need to summarize and refine them. Currently, complaint analysis relies primarily on manual methods, resulting in a large volume of data and low efficiency.
[0070] To address the aforementioned problems, this disclosure provides a data processing method.
[0071] Figure 1 A flowchart of a data processing method provided in an embodiment of this disclosure is shown.
[0072] like Figure 1 The methods shown may include:
[0073] S101, acquire data, wherein the data is data represented as text information;
[0074] S102, the data is clustered into preset categories according to the spectral clustering algorithm;
[0075] S103, in each category, a preset number of key data points and key data extracted according to the Textrank algorithm are used as a summary;
[0076] S104, Extract the specific summary corresponding to each category according to the preset model in each category;
[0077] S105, determine the general summary and specific summary of the current category as the target summary of the current category.
[0078] The data processing method provided in the embodiments of this disclosure acquires data, then clusters the data into preset categories using a spectral clustering algorithm. Within each category, a preset number of key data points and key data extracted using the Textrank algorithm are used as a general summary. Within each category, a specific summary corresponding to each category is extracted according to a preset model. The general summary and specific summary of the current category are then determined as the target summary for that category. Because the data is extracted, invalid data is eliminated. Furthermore, because the specific and general summaries of the data are acquired and determined as the target summary, the data volume is small while retaining its key elements, making the data easier to utilize and improving its applicability.
[0079] In S101, it should be noted that the data may include complaint data.
[0080] Data can be obtained from multiple channels and at multiple addresses. This disclosure does not specifically limit the channels and addresses for obtaining data.
[0081] In some implementations, data of other types besides text information can be acquired and converted into text data.
[0082] It should be noted that converting other types of data into text information is a common technique, which will not be described in detail here.
[0083] In S102, the preset category may include categories set by the user or data categories determined according to the amount of data.
[0084] It should be noted that clustering data using spectral clustering algorithms is a conventional algorithm, and this disclosure does not impose any specific limitations.
[0085] Spectral clustering can reduce the number of the most chaotic clusters in the clustering results.
[0086] S103 may specifically include:
[0087] Based on the TF-IDF algorithm, a predetermined number of key data points are extracted from the data in each category;
[0088] Extract key data from each category based on the TextRank algorithm;
[0089] A summary is obtained by integrating key data with important data.
[0090] It should be noted that multiple key data points can be extracted from the data in each category using the TF-IDF algorithm, and then these multiple key data points can be sorted. The top-ranked key data points, which are then selected as the final key data points.
[0091] The preset quantity can also be determined based on user customization.
[0092] A summary obtained by integrating key data with important data may include:
[0093] This involves fusing key data represented by textual information with important data represented by textual information.
[0094] In S104, it should be noted that the default model can be the UniLM+Textrank model.
[0095] Specifically, within each category, a specific summary corresponding to each category is extracted based on a preset model, including:
[0096] For each category, a summary corresponding to each category is extracted based on a supervised generative model;
[0097] The summaries corresponding to each category are merged to obtain the summary text for each category.
[0098] Input the summary text corresponding to each category into the summary extraction model to obtain the specific summary corresponding to each category.
[0099] It should be noted that the supervised generative model can be the UniLM model, and the extraction model can be the Textrank model.
[0100] In S105, it should be noted that determining the general summary and specific summary of the current category as the target summary of the current category can include:
[0101] The target summary is obtained by fusing the general summary identified by text information and the specific summary identified by text information.
[0102] For example, the general summaries are: business hall, account cancellation, and business, and the specific summary is: a telecommunications number was registered but not processed, resulting in outstanding fees.
[0103] The target summary is: business hall, account cancellation, and unresolved issues resulting in unpaid fees for previously registered telecom numbers.
[0104] The general summary is: quick payment, payment and outstanding fees; the specific summary is: quick payment 50 yuan monthly benefit package rebate details.
[0105] The target summary is: quick payment, payment and outstanding fees, and quick payment 50 yuan monthly benefit rebate.
[0106] In some embodiments, after S105, the data processing method may further include:
[0107] The target digest is sent to the user equipment so that the user can correct the target digest.
[0108] It should be noted that user equipment may include the user's corresponding terminal device, which may be various electronic devices, including but not limited to smartphones, tablets, laptops, desktop computers, wearable devices, augmented reality devices, virtual reality devices, etc.
[0109] For example, user modifications to the target summary may include:
[0110] The target summary is: Business hall, account cancellation, and outstanding fees due to unresolved issues with previously registered telecom numbers. The corrected summary for the user is: Business hall service processing.
[0111] The target summary was: Quick Pay, payment, outstanding fees, and the rebate of the 50 RMB monthly Quick Pay benefit package. The user-corrected summary is: Quick Pay fee issue.
[0112] In this embodiment of the disclosure, by sending the determined target digest to the user equipment so that the user can correct the target digest and obtain the user-corrected digest, the accuracy of the final determined digest can be improved.
[0113] In some embodiments, after S101 and before S102, the data processing method may further include:
[0114] The acquired data is filtered according to data standards to obtain the filtered data;
[0115] The filtered data is segmented into words according to the word segmentation criteria and algorithm to obtain segmented data.
[0116] Based on the synonym criteria, the synonyms in the segmented data are replaced with standard words to obtain the replaced data.
[0117] It should be noted that data standards can include user-defined standards based on the database.
[0118] For example, if the data standard is "Category...Remarks", then the data between the category and the remarks in the acquired data can be filtered out.
[0119] It should be noted that word segmentation criteria can include those determined by the user based on a standard thesaurus.
[0120] For example, a standard thesaurus may include: a telecommunications services thesaurus, a synonym thesaurus, and a place name thesaurus.
[0121] Word segmentation algorithms can include the Jieba algorithm.
[0122] For example, the synonym criteria can also be customized by the user.
[0123] In some implementations, after performing the above operations, the processed data can be further constrained using the TFIDF algorithm, such as by limiting the number of maximum feature vectors, the data ratio, and the frequency of occurrence.
[0124] In some implementations, PCA dimensionality reduction can be performed after transforming word vectors, and the random state of dimensionality reduction is set to 1 so that the result after each dimensionality reduction is the same.
[0125] In some embodiments, prior to S102, the data processing method may further include:
[0126] The data is clustered multiple times according to different numbers of categories to obtain the silhouette coefficients corresponding to each number of categories.
[0127] The number of categories corresponding to the largest silhouette coefficient is set to a preset number.
[0128] For example, if the data volume is between 0 and 1001, the number of clusters is determined to be 5, 6, 7, 8, 9, and 10, and then the silhouette coefficient corresponding to each number of clusters is obtained.
[0129] In some embodiments, prior to S104, the data processing method may further include:
[0130] The model is trained based on historical data, labeled text, and historical target text, and a preset model is obtained after the training stops.
[0131] It should be noted that historical data may include historical complaint data.
[0132] The annotation text can include the text being annotated.
[0133] Historical target text may include extracted text specified by the user.
[0134] For example, historical text may include: A user reported that they applied for a primary and secondary SIM card in August, and the local customer service informed them that the primary and secondary SIM cards could share the same package. In October, they discovered that additional charges were incurred without reason. The local customer service informed them that it was due to an operational error. The user did not accept this and requested expedited processing.
[0135] The annotation text may include: Customer service error caused additional charges to the primary and secondary cards.
[0136] Historical target text may include: customer charges resulting from customer service errors.
[0137] Historical texts may include: A user reported abnormal data usage between 7:00 AM and 11:00 AM on March 10th. The user then ordered a 10 RMB data package through the online store. The user claimed that the website did not clearly state that the data package was a 3-day data package, and that the user believed it was a full-month data package, and that the website's promotion was unclear.
[0138] Standard text may include: traffic packet, unclear.
[0139] Historical target text may include: unclear promotion of data packages.
[0140] Based on the same inventive concept, this disclosure also provides a data processing apparatus, as shown in the following embodiments. Since the principle by which this apparatus solves the problem is similar to that of the method embodiments described above, the implementation of this apparatus embodiment can refer to the implementation of the method embodiments described above, and repeated details will not be described again.
[0141] Figure 2 A schematic diagram of a data processing apparatus according to an embodiment of the present disclosure is shown.
[0142] like Figure 2 As shown, the device includes:
[0143] Acquisition module 201 is used to acquire data, wherein the data is represented as text information;
[0144] The first clustering module 202 is used to cluster the data into preset categories according to the spectral clustering algorithm;
[0145] The first extraction module 203 is used to extract a preset number of key data points and key data extracted according to the Textrank algorithm as a summary in each category;
[0146] The second extraction module 204 is used to extract the specific summary corresponding to each category according to the preset model in each category;
[0147] The first determining module 205 is used to determine the general summary and specific summary of the current category as the target summary of the current category.
[0148] The data processing apparatus provided in the embodiments of this disclosure acquires data, then clusters the data into preset categories using a spectral clustering algorithm. Within each category, a preset number of key data points and key data extracted using a Textrank algorithm are used as a general summary. Within each category, a specific summary corresponding to each category is extracted according to a preset model. The general summary and specific summary of the current category are then determined as the target summary for that category. Because the data is extracted, invalid data is eliminated. Furthermore, because the specific and general summaries of the data are acquired and determined as the target summary, the data volume is small while retaining its key elements, making the data easier to utilize and improving its applicability.
[0149] In one embodiment of this disclosure, the data processing apparatus further includes:
[0150] The sending module, after determining the general summary and specific summary of the current category as the target summary of the current category, sends the target summary to the user equipment so that the user can correct the target summary.
[0151] In one embodiment of this disclosure, the data processing apparatus further includes:
[0152] The filtering module, after acquiring data, including data represented by text information, and before clustering the data into preset categories according to the spectral clustering algorithm, is used to filter the acquired data according to data standards to obtain filtered data;
[0153] The word segmentation module is used to segment the filtered data into words according to the word segmentation criteria and algorithms, and obtain the segmented data.
[0154] The replacement module is used to replace synonyms in the segmented data with standard words based on the synonym criteria, so as to obtain the replaced data.
[0155] In one embodiment of this disclosure, the data processing apparatus further includes:
[0156] The second clustering module is used to cluster the data into preset categories according to the spectral clustering algorithm, and then cluster the data multiple times according to different numbers of categories to obtain the contour coefficients after clustering for each number of categories.
[0157] The second determining module is used to determine the number of categories corresponding to the largest contour coefficient as a preset number.
[0158] In one embodiment of this disclosure, the extraction module further includes:
[0159] The first extraction unit is used to extract a preset number of key data points from each category of data according to the TF-IDF algorithm.
[0160] The second extraction unit is used to extract key data for each category of data based on the Textrank algorithm;
[0161] The fusion unit is used to merge key data with important data to obtain a summary.
[0162] In one embodiment of this disclosure, the data processing apparatus further includes:
[0163] The training module, before extracting the specific summary corresponding to each category according to the preset model, is used to train the model based on historical data, labeled text, and historical target text. The preset model is obtained after the training stopping condition is met.
[0164] In one embodiment of this disclosure, the acquisition module includes:
[0165] The acquisition unit is used to acquire data of other types besides text information and convert other types of data into text type data.
[0166] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."
[0167] The following reference Figure 3 To describe an electronic device 300 according to such an embodiment of the present disclosure. Figure 3 The electronic device 300 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.
[0168] like Figure 3 As shown, the electronic device 300 is manifested in the form of a general-purpose computing device. The components of the electronic device 300 may include, but are not limited to: at least one processing unit 310, at least one storage unit 320, and a bus 330 connecting different system components (including storage unit 320 and processing unit 310).
[0169] The storage unit stores program code, which can be executed by the processing unit 310, causing the processing unit 310 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 310 can perform the following steps of the above method embodiments:
[0170] Acquire data, wherein the data is represented as text information;
[0171] The data is clustered into preset categories based on a spectral clustering algorithm;
[0172] In each category, a predetermined number of key data points and key data extracted according to the Textrank algorithm are used as a summary.
[0173] Extract specific summaries for each category based on a preset model;
[0174] The general summary and specific summary of the current category are determined as the target summary of the current category.
[0175] Storage unit 320 may include readable media in the form of volatile storage units, such as random access memory (RAM) 3201 and / or cache memory 3202, and may further include read-only memory (ROM) 3203.
[0176] Storage unit 320 may also include a program / utility 3203 having a set (at least one) program module 3205, such program module 3205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.
[0177] Bus 330 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.
[0178] Electronic device 300 can also communicate with one or more external devices 330 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 300, and / or with any device that enables electronic device 300 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 350. Furthermore, electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 360. As shown, network adapter 360 communicates with other modules of electronic device 300 via bus 330. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0179] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.
[0180] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the methods described above is stored thereon. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code, which, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.
[0181] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0182] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.
[0183] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.
[0184] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java and C++, and conventional procedural programming languages such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0185] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.
[0186] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.
[0187] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.
[0188] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.
Claims
1. A data processing method, characterized in that, The method includes: Acquire data, wherein the data is represented as text information; The data is clustered into preset categories based on a spectral clustering algorithm; In each category, a predetermined number of key data points and key data extracted according to the Textrank algorithm are used as a summary. Extract specific summaries for each category based on a preset model; The general summary and specific summary of the current category are determined as the target summary of the current category; Within each category, a predetermined number of key data points, along with key data extracted using the TextRank algorithm, are used as a summary, including: Based on the TF-IDF algorithm, a predetermined number of key data points are extracted from the data in each category; Extract key data from each category based on the TextRank algorithm; A summary is obtained by integrating key data with important data; Within each category, extract the specific summary corresponding to each category based on the preset model, including: For each category, a summary corresponding to each category is extracted based on a supervised generative model; The summaries corresponding to each category are merged to obtain the summary text for each category. Input the summary text corresponding to each category into the summary extraction model to obtain the specific summary corresponding to each category; Determining the general summary and specific summary of the current category as the target summary for the current category includes: The target summary is obtained by splicing and merging the general summary identified by text information and the specific summary identified by text information. After acquiring data, including data represented by text information, and before clustering the data into preset categories according to a spectral clustering algorithm, the method further includes: The acquired data is filtered according to data standards to obtain the filtered data; The filtered data is segmented into words according to the word segmentation criteria and algorithm to obtain segmented data. Based on the synonym criteria, the synonyms in the segmented data are replaced with standard words to obtain the replaced data.
2. The data processing method according to claim 1, characterized in that, After determining the general summary and specific summary of the current category as the target summary of the current category, the method further includes: The target digest is sent to the user equipment so that the user can correct the target digest.
3. The data processing method according to claim 1, characterized in that, Before clustering the data into preset categories according to the spectral clustering algorithm, the method further includes: The data is clustered multiple times according to different numbers of categories to obtain the silhouette coefficients corresponding to each number of categories. The number of categories corresponding to the largest silhouette coefficient is set to a preset number.
4. The data processing method according to claim 1, characterized in that, Before extracting the specific summary corresponding to each category based on a preset model in each category, the method further includes: The model is trained based on historical data, labeled text, and historical target text, and a preset model is obtained after the training stops.
5. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire data, wherein the data is represented as text information; The first clustering module is used to cluster the data into preset categories according to the spectral clustering algorithm; The first extraction module is used to extract a preset number of key data points and key data extracted according to the Textrank algorithm as a summary in each category; The second extraction module is used to extract the specific summary corresponding to each category based on a preset model. The first determining module is used to determine the general summary and specific summary of the current category as the target summary of the current category; Within each category, a predetermined number of key data points, along with key data extracted using the TextRank algorithm, are used as a summary, including: Based on the TF-IDF algorithm, a predetermined number of key data points are extracted from the data in each category; Extract key data from each category based on the TextRank algorithm; A summary is obtained by integrating key data with important data; Within each category, extract the specific summary corresponding to each category based on the preset model, including: For each category, a summary corresponding to each category is extracted based on a supervised generative model; The summaries corresponding to each category are merged to obtain the summary text for each category. Input the summary text corresponding to each category into the summary extraction model to obtain the specific summary corresponding to each category; Determining the general summary and specific summary of the current category as the target summary for the current category includes: The target summary is obtained by splicing and merging the general summary identified by text information and the specific summary identified by text information. After acquiring data, including data represented by text information, and before clustering the data into preset categories according to a spectral clustering algorithm, the process further includes: The acquired data is filtered according to data standards to obtain the filtered data; The filtered data is segmented into words according to the word segmentation criteria and algorithm to obtain segmented data. Based on the synonym criteria, the synonyms in the segmented data are replaced with standard words to obtain the replaced data.
6. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the data processing method of any one of claims 1 to 4 by executing the executable instructions.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data processing method according to any one of claims 1 to 4.