Data processing method and device based on ROUGE

Abstract

The invention provides a ROUGE-based data processing method and device. The ROUGE-based data processing method comprises the steps of obtaining a to-be-cleaned first data set; according to a preset splitting algorithm based on a ROUGE-L index, splitting the first data set to obtain a second data set and a third data set, and under the condition that the data in the second data set is greater than a first threshold value, performing semantic similarity calculation on the data in the second data set by adopting a BERT model to obtain a second threshold value; according to a second threshold value, splitting the second data set to obtain a fourth data set meeting a preset condition; and according to the third data set and the fourth data set, determining the third data set and the fourth data set as a target cleaning data set, by combining ROUGE-L and BERT algorithms, through multi-level similarity detection and dynamic data set division, realizing efficient and high-quality text data cleaning, and improving the accuracy and efficiency of cleaning.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a ROUGE-based data processing method and device. Background Art

[0002] In recent years, with the rapid development of natural language processing technology, the demand for large-scale, high-quality training data sets for high-quality text generation and understanding models has increased. However, existing data cleaning methods have problems such as incomplete processing, low efficiency, and a large impact on diversity. Simple deduplication and basic similarity detection techniques cannot completely remove duplicate or highly similar data, resulting in the training data set still being full of redundant information, which not only wastes computing resources, but may also cause model overfitting and affect the model's generalization ability. Summary of the invention

[0003] The present application shows a data processing method and device based on ROUGE.

[0004] In a first aspect, the present application shows a data processing method based on ROUGE, the method comprising:

[0005] Obtaining a first data set to be cleaned;

[0006] Splitting the first data set according to a preset splitting algorithm based on the ROUGE-L indicator to obtain a second data set and a third data set, wherein the data in the second data set is greater than the data in the third data set;

[0007] When the data in the second data set is greater than the first threshold, using the BERT model to perform semantic similarity calculation on the data in the second data set to obtain a second threshold;

[0008] Splitting the second data set according to the second threshold to obtain a fourth data set that meets a preset condition;

[0009] A target cleaned data set corresponding to the first data set to be cleaned is determined according to the third data set and the fourth data set.

[0010] Optionally, the step of splitting the first data set according to a preset evaluation algorithm to obtain a second data set and a third data set, wherein the data in the second data set is greater than the data in the third data set, comprises:

[0011] The first data set is split using the ROUGE-L evaluation algorithm;

[0012] Determine the data in the first data set that is greater than a first threshold as the second data set;

[0013] The data in the first data set that is less than or equal to the first threshold is determined as the third data set.

[0014] Optionally, the using the BERT model to perform semantic similarity calculation on the data in the second data set to obtain the second threshold includes:

[0015] Using the BERT model, calculate the cosine value between the sentences in the second dataset;

[0016] Obtain first N cosine values ​​of the cosine value, where N is a natural number greater than 0;

[0017] The second threshold is determined according to the average of the first N cosine values.

[0018] Optionally, splitting the second data set according to the second threshold to obtain a fourth data set that meets a preset condition includes:

[0019] The data in the second data set whose cosine similarity is less than the second threshold is determined as a fourth data set.

[0020] In a second aspect, the present application shows a data processing device based on ROUGE, the device comprising:

[0021] An acquisition module, used for acquiring a first data set to be cleaned;

[0022] An evaluation module, configured to split the first data set according to a preset splitting algorithm based on the ROUGE-L indicator to obtain a second data set and a third data set, wherein the data in the second data set is greater than the data in the third data set;

[0023] a calculation module, configured to, when the data in the second data set is greater than the first threshold, use the BERT model to perform semantic similarity calculation on the data in the second data set to obtain a second threshold;

[0024] a splitting module, configured to split the second data set according to the second threshold value to obtain a fourth data set that meets a preset condition;

[0025] The cleaning module is used to determine, based on the third data set and the fourth data set, a target cleaning data set corresponding to the first data set to be cleaned.

[0026] Optionally, the evaluation module is used to:

[0027] The first data set is split using the ROUGE-L evaluation algorithm;

[0028] Determine the data in the first data set that is greater than a first threshold as the second data set;

[0029] The data in the first data set that is less than or equal to the first threshold is determined as the third data set.

[0030] Optionally, the computing module is used to:

[0031] Using the BERT model, calculate the cosine value between the sentences in the second dataset;

[0032] Obtain first N cosine values ​​of the cosine value, where N is a natural number greater than 0;

[0033] The second threshold is determined according to the average of the first N cosine values.

[0034] Optionally, the splitting module is used to:

[0035] The data in the second data set whose cosine similarity is less than the second threshold is determined as a fourth data set.

[0036] In a third aspect, the present application shows an electronic device, which includes: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the method described in any of the above aspects.

[0037] In a fourth aspect, the present application illustrates a non-temporary computer-readable storage medium, which, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute a method as described in any of the above aspects.

[0038] In a fifth aspect, the present application illustrates a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in any of the above aspects.

[0039] The technical solution provided by the present application may include the following beneficial effects: A ROUGE-based data processing method and device provided in an embodiment of the present application include: obtaining a first data set to be cleaned; splitting the first data set according to a preset splitting algorithm based on the ROUGE-L indicator to obtain a second data set and a third data set, wherein the data in the second data set is greater than the data in the third data set; when the data in the second data set is greater than a first threshold, using a BERT model to perform semantic similarity calculation on the data in the second data set to obtain a second threshold; splitting the second data set according to the second threshold to obtain a fourth data set that meets preset conditions; determining, according to the third data set and the fourth data set, a target cleaned data set corresponding to the first data set to be cleaned. The embodiment of the present application combines the ROUGE-L and BERT algorithms, and realizes efficient and high-quality text data cleaning through multi-level similarity detection and dynamic data set division, thereby improving the accuracy and efficiency of cleaning. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a step flow chart of a data processing method based on ROUGE of the present application.

[0041] Figure 2 This is a flowchart of another ROUGE-based data processing method of the present application.

[0042] Figure 3 It is a structural block diagram of a ROUGE-based data processing device of the present application.

[0043] Figure 4 It is a block diagram of an electronic device of the present application.

[0044] Figure 5 It is a block diagram of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] Reference Figure 1 , shows a flow chart of the steps of a ROUGE-based data processing method of the present application, which can be applied to electronic devices, wherein the method specifically may include the following steps:

[0047] S101, obtaining a first data set to be cleaned;

[0048] S102, splitting the first data set according to a preset splitting algorithm based on the ROUGE-L indicator to obtain a second data set and a third data set, wherein the data in the second data set is greater than the data in the third data set;

[0049] S103. When the data in the second data set is greater than the first threshold, use the BERT model to calculate the semantic similarity of the data in the second data set to obtain a second threshold;

[0050] S104, splitting the second data set according to the second threshold value to obtain a fourth data set that meets a preset condition;

[0051] S105 . Determine, according to the third data set and the fourth data set, a target cleaning data set corresponding to the first data set to be cleaned.

[0052] Exemplarily, the present application embodiment provides a method for cleaning text data based on ROUGE improvement, comprising the following steps:

[0053] Step 1: Use ROUGE-L to divide the data set;

[0054] The ROUGE-L indicator is used to calculate the data set, and an F1 threshold (Value-1) is set. According to this threshold, the data set is divided into two subsets: data set A (the second data set) and data set B (the third data set), where data set A contains all samples whose F1 values ​​are greater than the first threshold Value-1, and the third database data set B contains all samples whose F1 values ​​are less than the threshold Value-1.

[0055] Step 2: Use the BERT algorithm to calculate semantic similarity;

[0056] The BERT algorithm is used to calculate sentence embedding for the samples in dataset A to obtain the embedding representation of each sentence. Then, the cosine similarity between the embeddings of each sentence is calculated, and the average of the first N similarities is taken as the new threshold Value-2. According to this threshold, i.e., the second threshold, dataset A is further divided into A1 and A2, i.e., the fourth dataset, where dataset A1 contains all samples whose mean cosine similarity is greater than the threshold Value-2, and dataset A2, i.e., the fourth dataset, contains all samples whose mean cosine similarity is less than the threshold Value-2.

[0057] Step 3: Eliminate redundant data;

[0058] Eliminate dataset A1 from the original dataset. In this way, high-quality cleaning of the dataset can be achieved. While removing redundant data, the diversity of samples is retained, thus ensuring that the dataset can provide richer information and better generalization performance during training.

[0059] A ROUGE-based data processing method provided in an embodiment of the present application includes: obtaining a first data set to be cleaned; splitting the first data set according to a preset splitting algorithm based on the ROUGE-L indicator to obtain a second data set and a third data set, wherein the data in the second data set is greater than the data in the third data set; when the data in the second data set is greater than a first threshold, using a BERT model to perform semantic similarity calculation on the data in the second data set to obtain a second threshold; splitting the second data set according to the second threshold to obtain a fourth data set that meets a preset condition; and determining, according to the third data set and the fourth data set, a target cleaned data set corresponding to the first data set to be cleaned. The embodiment of the present application combines the ROUGE-L and BERT algorithms, and realizes efficient and high-quality text data cleaning through multi-level similarity detection and dynamic data set division, thereby improving the accuracy and efficiency of cleaning.

[0060] Another embodiment of the present application further supplements the ROUGE-based data processing method provided in the above embodiment.

[0061] Optionally, according to a preset splitting algorithm based on the ROUGE-L indicator, the first data set is split to obtain a second data set and a third data set, wherein the data in the second data set is greater than the data in the third data set, including:

[0062] The ROUGE-L evaluation algorithm was used to split the first data set;

[0063] Determine the data in the first data set that is greater than the first threshold as the second data set;

[0064] The data in the first data set that is less than or equal to the first threshold is determined as the third data set.

[0065] Optionally, using a BERT model to calculate semantic similarity of data in the second data set to obtain a second threshold includes:

[0066] Using the BERT model, the cosine values ​​between the sentences in the second dataset are calculated;

[0067] Get the first N cosine values ​​among the cosine values, where N is a natural number greater than 0;

[0068] The second threshold is determined according to the average of the first N cosine values.

[0069] Optionally, the second data set is split according to the second threshold to obtain a fourth data set that meets a preset condition, including:

[0070] The data in the second data set whose cosine similarity is less than the second threshold is determined as the fourth data set.

[0071] like Figure 2 As shown, the embodiment of the present application discloses a method and device for cleaning text data based on ROUGE improvement, comprising the following steps:

[0072] Suppose there is a data set containing N pieces of text information:

[0073] 1. Dataset division:

[0074] Traverse the data set, arbitrarily combine two text messages, and use the ROUGE-L indicator to calculate the F1 mean between the two texts. Take the median of the F1 set as the threshold (Value-1). According to this threshold, the data set is divided into two subsets: data set A and data set B, where data set A contains all samples with F1 values ​​greater than the threshold Value-1, and data set B contains all samples with F1 values ​​less than the threshold Value-1.

[0075] 2. Use the BERT algorithm to calculate semantic similarity:

[0076] The BERT algorithm is used to calculate the sentence embedding for the samples in dataset A, and the embedding representation of each sentence is obtained. Then, the dataset is traversed, and the cosine similarity between the two sentence embeddings is calculated by arbitrarily combining two pieces of text information to obtain the set of all cosine similarities (k) in dataset A. And the mean of all values ​​greater than 0.5 in K is taken as the new threshold Value-2. According to this threshold, dataset A is further divided into A1 and A2, where dataset A1 contains all samples whose cosine similarity values ​​are greater than the threshold Value-2, and dataset A2 contains all samples whose cosine similarity values ​​are less than the threshold Value-2.

[0077] 3. Eliminate redundant data:

[0078] A1 is removed from the original data set, that is, data set A2 and data set B form the final cleaned data set.

[0079] Combine ROUGE-L and BERT algorithms for similarity detection: First, use ROUGE-L to preliminarily divide the data set, set the threshold Value-1 according to the F1 value, and divide the data set into data sets A and B, where A is the data set greater than the threshold Value-1.

[0080] (2) Use BERT to calculate sentence embedding representation: Apply the BERT algorithm to dataset A to calculate the embedding representation of each sentence, capture the deep semantic information of the text, and improve the accuracy of similarity detection.

[0081] (3) Dynamic dataset division: Based on the BERT embedding representation, the cosine similarity between adjacent sentences is calculated, and the mean of the top-N is taken as the threshold Value-2. The dataset A is further divided into A1 and A2, where A1 is the dataset greater than the threshold Value-2.

[0082] (4) Eliminate high-quality data and retain diversity: Eliminate dataset A1 from the dataset to ensure that while removing redundant data, the diversity and representativeness of the dataset are retained to prevent valuable information from being excessively deleted.

[0083] 2) Beneficial effects:

[0084] (1) Improve data cleaning effect: By combining ROUGE-L and BERT algorithms, redundant data can be detected and removed more accurately, ensuring the high quality and purity of the data set.

[0085] (2) Improve model performance: The cleaned data set is more representative and diverse, which significantly improves the generalization ability and recognition effect of the model, making the model perform better in various application scenarios.

[0086] (3) Improve processing efficiency: The multi-level and multi-threshold dynamic data set partitioning strategy improves the efficiency of data cleaning, reduces processing time and hardware resource requirements, and is suitable for efficient processing of large-scale data sets.

[0087] (4) Reduce costs: The automated data cleaning process reduces the need for manual intervention and reduces labor costs. It also reduces training time and hardware costs through efficient data processing.

[0088] (5) Enhanced scalability of the method: This method is based on the mature ROUGE-L and BERT algorithms, which can be easily integrated into the existing text processing and model training processes, facilitate practical application and promotion, and has good scalability and adaptability.

[0089] The embodiments of the present application are applicable to various fields that need to process large-scale text data, including but not limited to natural language processing, text generation, information retrieval, machine translation and other scenarios. For institutions that need to accumulate and use large amounts of data over a long period of time, such as Internet companies, data analysis companies and scientific research institutions, the present invention can significantly improve the level of data management and has broad application prospects and commercial value.

[0090] In summary, the embodiment of the present application combines ROUGE-L and BERT algorithms to achieve efficient and high-quality text data cleaning through multi-level similarity detection and dynamic data set division.

[0091] It should be noted that each implementable method in this embodiment may be implemented separately, or may be implemented in combination in any combination without conflict, and this application is not limited thereto.

[0092] A ROUGE-based data processing method provided in an embodiment of the present application includes: obtaining a first data set to be cleaned; splitting the first data set according to a preset splitting algorithm based on the ROUGE-L indicator to obtain a second data set and a third data set, wherein the data in the second data set is greater than the data in the third data set; when the data in the second data set is greater than a first threshold, using a BERT model to perform semantic similarity calculation on the data in the second data set to obtain a second threshold; splitting the second data set according to the second threshold to obtain a fourth data set that meets a preset condition; and determining, according to the third data set and the fourth data set, a target cleaned data set corresponding to the first data set to be cleaned. The embodiment of the present application combines the ROUGE-L and BERT algorithms, and realizes efficient and high-quality text data cleaning through multi-level similarity detection and dynamic data set division, thereby improving the accuracy and efficiency of cleaning.

[0093] Another embodiment of the present application provides a ROUGE-based data processing device, which is used to execute the ROUGE-based data processing method provided by the above embodiment.

[0094] like Figure 3 , which is a schematic diagram of the structure of a ROUGE-based data processing device provided in an embodiment of the present application. The ROUGE-based data processing device includes an acquisition module 301, an evaluation module 302, a calculation module 303, a splitting module 304 and a cleaning module 305, wherein:

[0095] The acquisition module 301 is used to acquire a first data set to be cleaned;

[0096] The evaluation module 302 is used to split the first data set according to a preset splitting algorithm based on the ROUGE-L indicator to obtain a second data set and a third data set, wherein the data in the second data set is greater than the data in the third data set;

[0097] The calculation module 303 is used to use the BERT model to perform semantic similarity calculation on the data in the second data set to obtain a second threshold value when the data in the second data set is greater than the first threshold value;

[0098] The splitting module 304 is used to split the second data set according to the second threshold value to obtain a fourth data set that meets the preset condition;

[0099] The cleaning module 305 is used to determine, based on the third data set and the fourth data set, a target cleaning data set corresponding to the first data set to be cleaned.

[0100] Regarding the device in this embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0101] A ROUGE-based data processing device provided in an embodiment of the present application includes: obtaining a first data set to be cleaned; splitting the first data set according to a preset splitting algorithm based on the ROUGE-L indicator to obtain a second data set and a third data set, wherein the data in the second data set is greater than the data in the third data set; when the data in the second data set is greater than a first threshold, using a BERT model to perform semantic similarity calculation on the data in the second data set to obtain a second threshold; splitting the second data set according to the second threshold to obtain a fourth data set that meets a preset condition; and determining, according to the third data set and the fourth data set, a target cleaned data set corresponding to the first data set to be cleaned. The embodiment of the present application combines the ROUGE-L and BERT algorithms, and realizes efficient and high-quality text data cleaning through multi-level similarity detection and dynamic data set division, thereby improving the accuracy and efficiency of cleaning.

[0102] Another embodiment of the present application further supplements the ROUGE-based data processing device provided in the above embodiment.

[0103] Optionally, an evaluation module is used to:

[0104] The ROUGE-L evaluation algorithm was used to split the first data set;

[0105] Determine the data in the first data set that is greater than the first threshold as the second data set;

[0106] The data in the first data set that is less than or equal to the first threshold is determined as the third data set.

[0107] Optionally, the computing module is used to:

[0108] Using the BERT model, the cosine values ​​between the sentences in the second dataset are calculated;

[0109] Get the first N cosine values ​​among the cosine values, where N is a natural number greater than 0;

[0110] The second threshold is determined according to the average of the first N cosine values.

[0111] Optionally, split the module to:

[0112] The data in the second data set whose cosine similarity is less than the second threshold is determined as the fourth data set.

[0113] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0114] A ROUGE-based data processing device provided in an embodiment of the present application includes: obtaining a first data set to be cleaned; splitting the first data set according to a preset splitting algorithm based on the ROUGE-L indicator to obtain a second data set and a third data set, wherein the data in the second data set is greater than the data in the third data set; when the data in the second data set is greater than a first threshold, using a BERT model to perform semantic similarity calculation on the data in the second data set to obtain a second threshold; splitting the second data set according to the second threshold to obtain a fourth data set that meets a preset condition; and determining, according to the third data set and the fourth data set, a target cleaned data set corresponding to the first data set to be cleaned. The embodiment of the present application combines the ROUGE-L and BERT algorithms, and realizes efficient and high-quality text data cleaning through multi-level similarity detection and dynamic data set division, thereby improving the accuracy and efficiency of cleaning.

[0115] Optionally, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0116] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0117] Figure 4 800 is a block diagram of an electronic device 800 shown in the present application. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0118] Reference Figure 4 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0119] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0120] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, images, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0121] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0122] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0123] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0124] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0125] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0126] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast operation information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0127] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0128] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by a processor 820 of an electronic device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0129] Figure 5 19 is a block diagram of a computer-readable storage medium 1900 shown in the present application. For example, the computer-readable storage medium 1900 may be provided as a server.

[0130] Reference Figure 5 , the computer-readable storage medium 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0131] The computer readable storage medium 1900 may also include a power supply component 1926 configured to perform power management of the computer readable storage medium 1900, a wired or wireless network interface 1950 configured to connect the computer readable storage medium 1900 to a network, and an input / output (I / O) interface 1958. The computer readable storage medium 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0132] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0133] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0134] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0135] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0136] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0137] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0138] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0139] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0140] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0141] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method based on ROUGE, characterized in that: The method comprises: Obtaining a first data set to be cleaned; Splitting the first data set according to a preset splitting algorithm based on the ROUGE-L indicator to obtain a second data set and a third data set, wherein the data in the second data set is greater than the data in the third data set; When the data in the second data set is greater than the first threshold, using the BERT model to perform semantic similarity calculation on the data in the second data set to obtain a second threshold; Splitting the second data set according to the second threshold to obtain a fourth data set that meets a preset condition; A target cleaned data set corresponding to the first data set to be cleaned is determined according to the third data set and the fourth data set.

2. The ROUGE-based data processing method according to claim 1, characterized in that: The step of splitting the first data set according to a preset splitting algorithm based on the ROUGE-L indicator to obtain a second data set and a third data set, wherein the data in the second data set is greater than the data in the third data set, comprises: The first data set is split using the ROUGE-L evaluation algorithm; Determine the data in the first data set that is greater than a first threshold as the second data set; The data in the first data set that is less than or equal to the first threshold is determined as the third data set.

3. The ROUGE-based data processing method according to claim 1, characterized in that: The using the BERT model to calculate the semantic similarity of the data in the second data set to obtain the second threshold includes: Using the BERT model, calculate the cosine value between the sentences in the second dataset; Obtain first N cosine values ​​of the cosine value, where N is a natural number greater than 0; The second threshold is determined according to the average of the first N cosine values.

4. The ROUGE-based data processing method according to claim 1, characterized in that: The step of splitting the second data set according to the second threshold to obtain a fourth data set that meets a preset condition includes: The data in the second data set whose cosine similarity is less than the second threshold is determined as a fourth data set.

5. A data processing device based on ROUGE, characterized in that: The device comprises: An acquisition module, used for acquiring a first data set to be cleaned; An evaluation module, configured to split the first data set according to a preset splitting algorithm based on the ROUGE-L indicator to obtain a second data set and a third data set, wherein the data in the second data set is greater than the data in the third data set; a calculation module, configured to, when the data in the second data set is greater than the first threshold, use the BERT model to perform semantic similarity calculation on the data in the second data set to obtain a second threshold; a splitting module, configured to split the second data set according to the second threshold value to obtain a fourth data set that meets a preset condition; The cleaning module is used to determine, based on the third data set and the fourth data set, a target cleaning data set corresponding to the first data set to be cleaned.

6. The ROUGE-based data processing device according to claim 5, characterized in that: The evaluation module is used to: use ROUGE-L An evaluation algorithm is used to split the first data set; Determine the data in the first data set that is greater than a first threshold as the second data set; The data in the first data set that is less than or equal to the first threshold is determined as the third data set.

7. The ROUGE-based data processing device according to claim 5, characterized in that: The computing module is used for: Using the BERT model, calculate the cosine value between the sentences in the second dataset; Obtain first N cosine values ​​of the cosine value, where N is a natural number greater than 0; The second threshold is determined according to the average of the first N cosine values.

8. The ROUGE-based data processing device according to claim 5, characterized in that: The splitting module is used for: The data in the second data set whose cosine similarity is less than the second threshold is determined as a fourth data set.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 4 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images