Method, device and system for processing text data

Abstract

The present disclosure provides a method, device, and system for processing text data. The method includes: adding a text processing operator for processing text data to an operator set of a machine learning platform, wherein the text processing operator includes: a summary extraction operator and / or a text similarity analysis operator; and performing text data processing based on the text processing operator in the machine learning platform.

Description

Technical Field

[0001] The present invention relates to the technical field of text data processing, and more specifically, to a text data processing method, a text data processing device, a system including at least one computing device and at least one storage device, and a readable storage medium. Background Art

[0002] With the development of artificial intelligence, the value of data continues to grow. The need to automatically output article summaries and compare similarities between two articles using natural language processing technology is becoming increasingly common. These needs primarily involve the training, prediction, and deployment of corresponding text processing models.

[0003] However, training, prediction, and deployment of text processing models require the expertise of natural language processing professionals. Without natural language processing experience, it's difficult for those with this experience to implement these capabilities in real-world scenarios. Consequently, the limited capabilities of processing personnel significantly limit the practical application of article summarization and text similarity analysis in real-world scenarios, leading to the waste and inability to utilize vast amounts of valuable text data. Summary of the Invention

[0004] One purpose of the present disclosure is to provide a new technical solution for processing text data.

[0005] According to a first aspect of the present disclosure, a method for processing text data is provided, comprising:

[0006] Adding a text processing operator for processing text data to the operator set of the machine learning platform, wherein the text processing operator includes: a summary extraction operator and / or a text similarity analysis operator;

[0007] Text data processing is performed based on the text processing operator in the machine learning platform.

[0008] Optionally, the performing text data processing based on the text processing operator in the machine learning platform includes:

[0009] Providing a user configuration interface, the user configuration interface including an operator display area and a canvas area for creating a data processing flow chart; wherein the operator display area displays the text processing operator for processing text data;

[0010] Get the uploaded target text data;

[0011] In response to an operation of creating a target data processing flowchart, creating the target data processing flowchart in the canvas area according to the target text data and the text processing operator;

[0012] In response to the operation of running the target data processing flowchart, the text processing operator is run according to the target data processing flowchart to process the target text data and obtain a text processing result.

[0013] Optionally, in response to the operation of creating a target data processing flowchart, creating the target data processing flowchart in the canvas area according to the target text data and the text processing operator includes:

[0014] In response to an operation of selecting the target text data, displaying the target text data in the canvas area;

[0015] In response to an operation of selecting the text processing operator in the operator display area, displaying the text processing operator in the canvas area;

[0016] In response to the operation of connecting the target text data and the text processing operator, the target text data and the text processing operator are connected in the canvas area to obtain the target data processing flow chart.

[0017] Optionally, also include:

[0018] In response to a request for resource configuration of the text processing operator, providing a first configuration interface of the text processing operator;

[0019] The resource configuration parameters of the text processing operator are obtained through the first configuration interface to run the text processing operator according to the resource configuration parameters.

[0020] Optionally, obtaining the uploaded target text data includes:

[0021] Provide an entry for uploading data;

[0022] In response to an operation of uploading target text data, the target text data uploaded through the portal is acquired.

[0023] Optionally, the text processing operator is a summary extraction operator for performing summary extraction processing on text data;

[0024] The step of running the text processing operator to process the target text data according to the target data processing flow chart to obtain a text processing result includes:

[0025] Run the summary extraction operator to process the target text data as follows:

[0026] Performing sentence segmentation processing on target line text data in the target text data to obtain multiple sentences; wherein the target line text data is text data of any line in the target text data;

[0027] Determine the similarity between each two sentences, and perform regularization on the similarity between each two sentences to obtain a similarity matrix;

[0028] Obtaining scores of other sentences based on the similarity matrix and the preset scores of the specified sentence; wherein the other sentences are sentences other than the specified sentence in the plurality of sentences;

[0029] selecting a set number of sentences from the plurality of sentences as summary sentences based on the scores;

[0030] According to the summary sentence, a text summary corresponding to the target row of text data is generated as the text processing result.

[0031] Optionally, determining the similarity between each two sentences includes:

[0032] Encode each sentence to obtain the sentence vector of each sentence;

[0033] For every two sentences, the cosine value between the corresponding sentence vectors is determined as the similarity between the two sentences.

[0034] Optionally, generating a text summary corresponding to the target row of text data according to the summary sentence as the text processing result includes:

[0035] Obtaining the order of each summary sentence in the target row of text data;

[0036] The summary sentences are sorted in the order, and preset punctuation is added after each summary sentence to obtain a text summary corresponding to the target row of text data as the text processing result.

[0037] Optionally, the text processing operator is a text similarity analysis operator for performing similarity analysis on text data;

[0038] The step of running the text processing operator to process the target file data according to the target data processing flow chart to obtain a text processing result includes:

[0039] Run the text similarity analysis operator to process the target text data as follows:

[0040] Encoding first text data and second text data located in the same row of the target text data respectively to obtain a first vector of the first text data and a second vector of the second text data; wherein the first text data is located in a first target column of the target text data, and the second text data is located in a second target column of the target text data;

[0041] According to the first vector and the second vector, a predicted similarity between the first text data and the second text data is determined as the text processing result.

[0042] Optionally, determining the predicted similarity between the first text data and the second text data based on the first vector and the second vector includes:

[0043] determining a cosine similarity between the first vector and the second vector;

[0044] determining a maximum of the squares of the first vector and the second vector;

[0045] determining an absolute value of a difference between the first vector and the second vector;

[0046] determining a dot product of the first vector and the second vector;

[0047] Splicing the cosine similarity, the maximum value of the square, the absolute value of the difference, the dot product, the first vector, and the second vector to obtain a target splicing vector of the first text data and the second text data;

[0048] Obtain the mapping function between the splicing vector and the similarity;

[0049] Determine a predicted similarity between the first text data and the second text data according to the mapping function and the target concatenation vector.

[0050] Optionally, also include:

[0051] In response to a request for configuring a processing object for the text similarity analysis operator, providing a second configuration interface for the text similarity analysis operator;

[0052] The first target column and the second target column are obtained through the second configuration interface, so that the text similarity analysis operator performs similarity analysis on the text data of the first target column and the text data of the second target column in the target text data.

[0053] Optionally, also include:

[0054] The text processing result is displayed.

[0055] Optionally, the performing text data processing based on the text processing operator in the machine learning platform includes:

[0056] In response to a request to bring the text processing operator online, the text processing operator is packaged and processed to obtain a target estimation service for use by a target user;

[0057] The target estimation service is run to process text data provided by the target user to the target estimation service, and a corresponding text processing result is returned to the target user.

[0058] Optionally, in response to the request to bring the text processing operator online, packaging the text processing operator to obtain the target estimation service includes:

[0059] In response to a request to create an estimation service, providing at least one estimation module for user selection;

[0060] Obtain the target estimation module selected by the user and provide a model selection interface;

[0061] Obtaining the text processing operator selected by the user through the model selection interface;

[0062] According to the target estimation module, the text processing operators are packaged to obtain the target estimation service.

[0063] Optionally, also include:

[0064] In response to a request to configure the target estimation service, providing a third configuration interface;

[0065] The configuration information of the target estimation service is obtained through the third configuration interface, so as to run the target estimation service according to the configuration information.

[0066] Optionally, the configuration information includes one of the following: GPU resources used, CPU resources used, memory resources used, and the address of the image called at runtime.

[0067] Optionally, also include:

[0068] In response to a request to view the status of the target estimation service, the status of the target estimation service is displayed.

[0069] Optionally, the status of the target estimation service includes at least one of the following: resources occupied by the target estimation service during operation, the online history of the target estimation service, the version number of the target estimation service, the deployment time of the target estimation service, and the running time of the target estimation service.

[0070] According to a second aspect of the present disclosure, there is provided a device for processing text data, comprising:

[0071] An operator adding module, configured to add a text processing operator for processing text data to the operator set of the machine learning platform, wherein the text processing operator includes: a summary extraction operator and / or a text similarity analysis operator;

[0072] A text processing module is used to process text data based on the text processing operator in the machine learning platform.

[0073] Optionally, the text processing module is further configured to:

[0074] Providing a user configuration interface, the user configuration interface including an operator display area and a canvas area for creating a data processing flow chart; wherein the operator display area displays the text processing operator for processing text data;

[0075] Get the uploaded target text data;

[0076] In response to an operation of creating a target data processing flowchart, creating the target data processing flowchart in the canvas area according to the target text data and the text processing operator;

[0077] In response to the operation of running the target data processing flowchart, the text processing operator is run according to the target data processing flowchart to process the target text data and obtain a text processing result.

[0078] Optionally, in response to the operation of creating a target data processing flowchart, creating the target data processing flowchart in the canvas area according to the target text data and the text processing operator includes:

[0079] In response to an operation of selecting the target text data, displaying the target text data in the canvas area;

[0080] In response to an operation of selecting the text processing operator in the operator display area, displaying the text processing operator in the canvas area;

[0081] In response to the operation of connecting the target text data and the text processing operator, the target text data and the text processing operator are connected in the canvas area to obtain the target data processing flow chart.

[0082] Optionally, also include:

[0083] A module for providing a first configuration interface for the text processing operator in response to a request for resource configuration for the text processing operator;

[0084] A module for obtaining resource configuration parameters of the text processing operator through the first configuration interface to run the text processing operator according to the resource configuration parameters.

[0085] Optionally, obtaining the uploaded target text data includes:

[0086] Provide an entry for uploading data;

[0087] In response to an operation of uploading target text data, the target text data uploaded through the portal is acquired.

[0088] Optionally, the text processing operator is a summary extraction operator for performing summary extraction processing on text data;

[0089] The step of running the text processing operator to process the target text data according to the target data processing flow chart to obtain a text processing result includes:

[0090] Run the summary extraction operator to process the target text data as follows:

[0091] Performing sentence segmentation processing on target line text data in the target text data to obtain multiple sentences; wherein the target line text data is text data of any line in the target text data;

[0092] Determine the similarity between each two sentences, and perform regularization on the similarity between each two sentences to obtain a similarity matrix;

[0093] Obtaining scores of other sentences based on the similarity matrix and the preset scores of the specified sentence; wherein the other sentences are sentences other than the specified sentence in the plurality of sentences;

[0094] selecting a set number of sentences from the plurality of sentences as summary sentences based on the scores;

[0095] According to the summary sentence, a text summary corresponding to the target row of text data is generated as the text processing result.

[0096] Optionally, determining the similarity between each two sentences includes:

[0097] Encode each sentence to obtain the sentence vector of each sentence;

[0098] For every two sentences, the cosine value between the corresponding sentence vectors is determined as the similarity between the two sentences.

[0099] Optionally, generating a text summary corresponding to the target row of text data according to the summary sentence as the text processing result includes:

[0100] Obtaining the order of each summary sentence in the target row of text data;

[0101] The summary sentences are sorted in the order, and preset punctuation is added after each summary sentence to obtain a text summary corresponding to the target row of text data as the text processing result.

[0102] Optionally, the text processing operator is a text similarity analysis operator for performing similarity analysis on text data;

[0103] The step of running the text processing operator to process the target file data according to the target data processing flow chart to obtain a text processing result includes:

[0104] Run the text similarity analysis operator to process the target text data as follows:

[0105] Encoding first text data and second text data located in the same row of the target text data respectively to obtain a first vector of the first text data and a second vector of the second text data; wherein the first text data is located in a first target column of the target text data, and the second text data is located in a second target column of the target text data;

[0106] According to the first vector and the second vector, a predicted similarity between the first text data and the second text data is determined as the text processing result.

[0107] Optionally, determining the predicted similarity between the first text data and the second text data based on the first vector and the second vector includes:

[0108] determining a cosine similarity between the first vector and the second vector;

[0109] determining a maximum of the squares of the first vector and the second vector;

[0110] determining an absolute value of a difference between the first vector and the second vector;

[0111] determining a dot product of the first vector and the second vector;

[0112] Splicing the cosine similarity, the maximum value of the square, the absolute value of the difference, the dot product, the first vector, and the second vector to obtain a target splicing vector of the first text data and the second text data;

[0113] Obtain the mapping function between the splicing vector and the similarity;

[0114] Determine a predicted similarity between the first text data and the second text data according to the mapping function and the target concatenation vector.

[0115] Optionally, also include:

[0116] A module for providing a second configuration interface of the text similarity analysis operator in response to a request for configuring a processing object of the text similarity analysis operator;

[0117] A module for obtaining the first target column and the second target column through the second configuration interface, so that the text similarity analysis operator performs similarity analysis on the text data of the first target column and the text data of the second target column in the target text data.

[0118] Optionally, also include:

[0119] A module for displaying the text processing results.

[0120] Optionally, the text processing module is further configured to:

[0121] In response to a request to bring the text processing operator online, the text processing operator is packaged and processed to obtain a target estimation service for use by a target user;

[0122] The target estimation service is run to process text data provided by the target user to the target estimation service, and a corresponding text processing result is returned to the target user.

[0123] Optionally, in response to the request to bring the text processing operator online, packaging the text processing operator to obtain the target estimation service includes:

[0124] In response to a request to create an estimation service, providing at least one estimation module for user selection;

[0125] Obtain the target estimation module selected by the user and provide a model selection interface;

[0126] Obtaining the text processing operator selected by the user through the model selection interface;

[0127] According to the target estimation module, the text processing operators are packaged to obtain the target estimation service.

[0128] Optionally, also include:

[0129] A module for providing a third configuration interface in response to a request to configure the target estimation service;

[0130] A module for obtaining configuration information of the target estimation service through the third configuration interface, so as to run the target estimation service according to the configuration information.

[0131] Optionally, the configuration information includes one of the following: GPU resources used, CPU resources used, memory resources used, and the address of the image called at runtime.

[0132] Optionally, also include:

[0133] Means for displaying a status of the target estimation service in response to a request to view the status of the target estimation service.

[0134] Optionally, the status of the target estimation service includes at least one of the following: resources occupied by the target estimation service during operation, the online history of the target estimation service, the version number of the target estimation service, the deployment time of the target estimation service, and the running time of the target estimation service.

[0135] According to a third aspect of the present disclosure, a system comprising at least one computing device and at least one storage device is provided, wherein the at least one storage device is used to store instructions, and the instructions are used to control the at least one computing device to execute the method according to the first aspect of the present disclosure.

[0136] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0137] The data processing method of the disclosed embodiments pre-introduces a text processing operator to the set of operators in the machine learning platform. This operator can then be directly called within the machine learning platform to process text data, eliminating the need to rebuild the corresponding text processing workflow. This allows users to complete text data processing without requiring specialized natural speech processing knowledge or scenario-specific experience, resulting in truly out-of-the-box functionality and lowering the barrier to entry.

[0138] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0139] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0140] Figure 1 is a block diagram of an example of a hardware configuration of an electronic device that can be used to implement an embodiment of the present disclosure.

[0141] Figure 2 is a flowchart of a method for processing text data according to an embodiment of the present disclosure;

[0142] Figure 3 is a flowchart of an example of a method for processing text data according to an embodiment of the present disclosure;

[0143] Figure 4 is a schematic diagram of uploading target text data according to an embodiment of the present disclosure;

[0144] Figure 5 is a flowchart of another example of a method for processing text data according to an embodiment of the present disclosure;

[0145] Figure 6 is a schematic diagram of a target data processing flow chart according to an embodiment of the present disclosure;

[0146] Figure 7 is a schematic diagram of a selection interface for an estimation service according to an embodiment of the present disclosure;

[0147] Figure 8 is a schematic diagram of a model selection interface according to an embodiment of the present disclosure;

[0148] Figure 9 is a block diagram of a text data processing apparatus according to an embodiment of the present disclosure;

[0149] Figure 10 is a block diagram of a system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0150] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0151] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0152] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0153] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0154] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0155] Hereinafter, various embodiments and examples according to embodiments of the present invention will be described with reference to the accompanying drawings.

[0156] <Hardware Configuration>

[0157] Figure 1 is a block diagram illustrating a hardware configuration of an electronic device 1000 that can implement an embodiment of the present disclosure.

[0158] The electronic device 1000 may be a portable computer, a desktop computer, a mobile phone, a tablet computer, etc. Figure 1 As shown, electronic device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and the like. Processor 1100 may be a central processing unit (CPU), a microprocessor (MCU), or the like. Memory 1200 may include, for example, ROM (read-only memory), RAM (random access memory), or a non-volatile memory such as a hard disk. Interface device 1300 may include, for example, a USB interface or a headphone jack. Communication device 1400 may be capable of wired or wireless communication, specifically, Wi-Fi, Bluetooth, 2G / 3G / 4G / 5G communication, or the like. Display device 1500 may be, for example, an LCD display or a touchscreen display. Input device 1600 may include, for example, a touchscreen, a keyboard, or somatosensory input. Users may input and output voice information through speaker 1700 and microphone 1800.

[0159] Figure 1 The electronic device shown is merely illustrative and does not in any way limit the present disclosure, its application or use. In the embodiments of the present disclosure, the memory 1200 of the electronic device 1000 is used to store instructions, which are used to control the processor 1100 to operate to perform any of the methods provided in the embodiments of the present disclosure. It should be understood by those skilled in the art that although Figure 1 While multiple devices are shown for electronic device 1000, the present disclosure may only relate to some of these devices. For example, electronic device 1000 may only relate to processor 1100 and storage device 1200. A skilled person can design instructions based on the solutions disclosed in this disclosure. How instructions control processor operations is well known in the art and will not be described in detail here.

[0160] <Method Example>

[0161] <Example 1>

[0162] In this embodiment, a method for processing text data is provided. The method for processing text data can be implemented by an electronic device. The electronic device can be Figure 1 The electronic device 1000 is shown.

[0163] according to Figure 2 As shown, the text data processing method of this embodiment may include the following steps S2100 to S2200:

[0164] Step S2100: Add a text processing operator for processing text data to the operator set of the machine learning platform.

[0165] In one embodiment of the present disclosure, the text processing operator may include a summary extraction operator and / or a text similarity analysis operator.

[0166] In an embodiment where the text processing operator includes a summary extraction operator, the summary extraction operator can be used to perform summary extraction processing on text data to obtain a summary of the text data.

[0167] In an embodiment where the text processing operator includes a text similarity analysis operator, the text similarity analysis operator can be used to perform similarity analysis on the first text data and the second text data to obtain the similarity between the first text data and the second text data.

[0168] The operators provided in the operator set of the machine learning platform in this embodiment can be directly called by the machine learning platform.

[0169] Step S2200: Perform text data processing based on the text processing operator in the machine learning platform.

[0170] The data processing method of the disclosed embodiments pre-introduces a text processing operator to the set of operators in the machine learning platform. This operator can then be directly called within the machine learning platform to process text data, eliminating the need to rebuild the corresponding text processing workflow. This allows users to complete text data processing without requiring specialized natural speech processing knowledge or scenario-specific experience, resulting in truly out-of-the-box functionality and lowering the barrier to entry.

[0171] <Example 2>

[0172] Based on the above-mentioned embodiment 1, this embodiment provides a method for processing text data based on a text processing operator in a machine learning platform, which may specifically include the following: Figure 3 Steps S3100 to S3400 are shown:

[0173] Step S3100: providing a user configuration interface, which includes an operator display area and a canvas area.

[0174] The operator display area of ​​this embodiment displays text processing operators for processing text data, including summary extraction operators and / or text similarity analysis operators. The canvas area of ​​this embodiment can be used to create a data processing flow chart based on the operators displayed in the operator display area.

[0175] A data processing flow chart can be used to represent a corresponding data processing process. In one embodiment, the data processing flow chart is a directed acyclic graph (DAG). A directed acyclic graph (DAG) is a directed graph without loops.

[0176] Step S3200: Obtain uploaded target text data.

[0177] The target text data in this embodiment may be pre-uploaded to the machine learning platform. When executing step S3200, the target text data in the machine learning platform may be directly called.

[0178] In one embodiment of the present disclosure, obtaining the uploaded target text data may further include:

[0179] An entry for uploading data is provided; in response to an operation of uploading target text data, the target text data uploaded through the entry is obtained.

[0180] For example, Figure 4 As shown, methods for uploading target text data are provided, including local upload (supporting CSV, TSV, TXT, Parquet, and ORC formats), importing from FTP (supporting CSV, TSV, and TXT formats), importing from HDFS (supporting CSV, TSV, TXT, Parquet, and ORC formats), shallow copying from HDFS (supporting Parquet format), importing from databases (supporting Oracle, MySQL, Teradata, Vertica, and other databases), and importing from Hive (supporting Simple Auth or Kerberos authentication). Users can select an upload method based on their needs and click the "Click to Upload" button, which is the entry point for uploading data. This button provides data that can be uploaded to the machine learning platform. Users can then select the target text data to upload based on their needs, triggering the upload operation.

[0181] Step S3300 , in response to the operation of creating a target data processing flowchart, a target data processing flowchart is created in the canvas area according to the target text data and the text processing operator.

[0182] In one embodiment of the present disclosure, in response to the operation of creating a target data processing flowchart, creating the target data processing flowchart in the canvas area according to the target text data and the text processing operator may include the following steps S3310 to S3330:

[0183] Step S3310: In response to the operation of selecting target text data, the target text data is displayed in the canvas area.

[0184] In one embodiment of the present disclosure, the operation of selecting the target text data may be the aforementioned operation of uploading the target text data. By uploading the target text data, the target text data may be directly displayed in the canvas area.

[0185] In another embodiment of the present disclosure, the user configuration interface may further include a data display area. When the user uploads the target text data to the machine learning platform in advance, the data display area may display the target text data.

[0186] On the basis of this embodiment, the operation of selecting the target text data can be an operation of dragging the target text data in the data display area to the canvas area. By dragging the target text data in the data display area to the canvas area, the user can trigger the electronic device to display the dragged target text data in the canvas area.

[0187] Based on this embodiment, the operation of selecting the target text data can also be performed on the target text data in the data display area. The selection operation can include a double-click operation, a single-click operation, or an operation of right-clicking and clicking a selection button. By performing the selection operation on the target text data in the data display area, the user can display the target text data in the canvas area.

[0188] Step S3320 : In response to the operation of selecting the text processing operator in the operator display area, the text processing operator is displayed in the canvas area.

[0189] In one embodiment of the present disclosure, the operation of selecting a text processing operator in the operator display area may be an operation of dragging the text processing operator in the operator display area to the canvas area. By dragging the text processing operator in the operator display area to the canvas area, the user may trigger the electronic device to display the dragged text processing operator in the canvas area.

[0190] In another embodiment of the present disclosure, the operation of selecting a text processing operator in the operator display area may also be performing a selection operation on the text processing operator in the operator display area. The selection operation may include a double-click operation, a single-click operation, or a right-click and a selection button operation. By performing a selection operation on the text processing operator in the operator display area, the user may display the text processing operator in the canvas area.

[0191] Step S3330 , in response to the operation of connecting the target text data and the text processing operator, the target text data and the text processing operator are connected in the canvas area to obtain a target data processing flow chart.

[0192] In one embodiment of the present disclosure, in the canvas area, the target text data has a downstream connection point for representing the output of the data; the text processing operator may have at least an upstream connection point for representing the input of the data.

[0193] The user can trigger the operation of connecting the target text data and the text processing operator in the canvas area by clicking the downstream connection point of the target text data and the upstream connection point of the text processing operator respectively. In response to the operation, the electronic device connects the target text data and the text processing operator displayed in the canvas area.

[0194] In another embodiment of the present disclosure, the user may place a text processing operator below the target text data in the canvas area, triggering an operation of connecting the target text data and the text processing operator. In response to the operation, the electronic device automatically connects the downstream connection point of the target text data displayed in the canvas area and the upstream connection point of the text processing operator.

[0195] In one embodiment of the present disclosure, the target text data and the text processing operator are connected in the canvas area, and the target data processing flow chart obtained can be as follows: Figure 6 shown.

[0196] In one embodiment of the present disclosure, the method may further include:

[0197] In response to a request for resource configuration of a text processing operator, a first configuration interface of the text processing operator is provided; resource configuration parameters of the text processing operator are obtained through the first configuration interface to run the text processing operator according to the resource configuration parameters.

[0198] The resource configuration parameters in this embodiment may include CPU usage, GPU usage, memory usage, etc.

[0199] Step S3400 , in response to the operation of running the target data processing flowchart, runs the text processing operator according to the target data processing flowchart to process the target text data and obtains a text processing result.

[0200] In an embodiment where the text processing operator is a summary extraction operator, in response to the operation of executing the target data processing flowchart, executing the text processing operator according to the target data processing flowchart to process the target text data to obtain the text processing result may include: executing the summary extraction operator to perform the following processing steps S3410 to S3450 on the target text data:

[0201] Step S3410 , segmenting the target line of text data in the target text data to obtain multiple sentences.

[0202] The target row text data is the text data of any row in the target text data.

[0203] In one embodiment of the present disclosure, the target text data may be structured data, and each line of data may represent a complete article. Specifically, each line of data of the target text data may be traversed to obtain the target line text data.

[0204] The target line text data may be segmented according to preset punctuation marks, and the portion between any two adjacent preset punctuation marks may be regarded as a sentence.

[0205] Step S3420 , determining the similarity between every two sentences, and performing regularization processing on the similarity between every two sentences to obtain a similarity matrix.

[0206] In one embodiment of the present disclosure, determining the similarity between each two sentences may include steps S3421 to S3422 as follows:

[0207] Step S3421, encode each sentence to obtain a sentence vector for each sentence.

[0208] In one embodiment of the present disclosure, each sentence may be encoded separately through an embedding model to obtain a sentence vector of the corresponding sentence.

[0209] Step S3422: For every two sentences, determine the cosine value between the corresponding sentence vectors as the similarity between the two sentences.

[0210] For example, for the first and second sentences, the sentence vector of the first sentence can be expressed as A, and the sentence vector of the second sentence can be expressed as B, then the similarity S between the first and second sentences is 1,2 It can be expressed as:

[0211]

[0212] When obtaining the similarity between each two sentences, a similarity matrix may be obtained according to the similarity between each two sentences.

[0213] Specifically, in the similarity matrix, the element corresponding to the i-th row and j-th column can be the similarity between the i-th statement and the j-th statement, where i, j∈[1, N], and N is the number of multiple statements obtained by segmenting the target row statement.

[0214] Step S3430: Obtain scores of other sentences based on the similarity matrix and the preset scores of the specified sentence.

[0215] Among them, other statements are statements other than the specified statement in the multiple statements.

[0216] In one embodiment of the present disclosure, the designated statement may be the nth statement pre-set based on an application scenario or specific requirements, where n∈[1,N], where N is the number of statements obtained by segmenting the target line of statements. That is, n is pre-set by the user based on the application scenario or specific requirements. For example, if n is 1, then the first statement is the designated statement.

[0217] The preset score of the specified statement can be set in advance according to the application scenario or specific needs. For example, the preset score of the specified statement can be 1.

[0218] Generally, the first sentence of an article, that is, the first statement is a more important statement. Therefore, the first statement can be used as the designated statement.

[0219] In one embodiment of the present disclosure, based on the similarity matrix and the preset score of the nth statement (specified statement), the score f(k) of the kth statement can be expressed as:

[0220] f(k)=α*S*f(k-1)+(1-α)*y

[0221] Wherein, k is an integer other than n in [1, N], S is the similarity matrix, α is a preset parameter greater than 0 and less than 1, and f(k-1) is the score of the k-1th statement.

[0222] y is a vector representing a specified statement. The value of the nth element in the vector is a first set value, and the values ​​of the remaining elements are second set values. The first set value and the second set value are different values ​​set according to the application scenario or specific requirements. For example, the first set value can be 1 and the second set value can be 0.

[0223] For example, if the first statement is a designated statement, y can be expressed as [1, 0, ... 0, 0]. If the second statement is a designated statement, y can be expressed as [0, 1, ... 0, 0]. If the Nth statement is a designated statement, y can be expressed as [0, 0, ... 0, N].

[0224] Step S3440 : Select a set number of sentences from the multiple sentences as summary sentences based on the scores.

[0225] In one embodiment of the present disclosure, a set number of statements with the highest scores may be selected as summary statements. Specifically, the statements may be sorted in descending order based on their scores, and the ranking value of each statement may be recorded. Statements with ranking values ​​less than or equal to the set number may be selected as summary statements.

[0226] Step S3450: Generate a text summary corresponding to the target row of text data according to the summary sentence as a text processing result.

[0227] In one embodiment of the present disclosure, generating a text summary corresponding to the target row of text data according to the summary sentence as a text processing result may include the following steps S3451 to S3452:

[0228] Step S3451: Obtain the order of each summary sentence in the target row of text data.

[0229] Step S3452 , sorting the summary sentences in order, and adding preset punctuation after each summary sentence, to obtain a text summary corresponding to the target row of text data as a text processing result.

[0230] In this embodiment, the summary sentences may be sorted according to the order in the target row text data, and a preset punctuation mark may be added after each summary sentence to obtain the corresponding text summary.

[0231] The punctuation added after each summary sentence may be pre-set according to the application scenario or specific requirements, or may be determined according to the punctuation of the corresponding summary sentence in the target row of text data.

[0232] In an embodiment where the punctuation added after each summary statement is pre-set based on the application scenario or specific needs, the punctuation added after each summary statement can be the same or different. For example, the punctuation added after all but the last summary statement can be a comma, and the punctuation added after the last summary statement can be a period or an exclamation mark.

[0233] In an embodiment in which the punctuation added after each summary sentence is set based on the punctuation marks in the target line of text data, the punctuation marks except for the last summary sentence in the target line of text data may be added after the corresponding summary sentence. If the punctuation marks in the last summary sentence in the target line of text data are other than commas, the punctuation marks in the last summary sentence in the target line of text data may be added after the summary sentence. If the punctuation marks in the last summary sentence in the target line of text data are commas, the punctuation marks added after the last summary sentence may be a period or an exclamation mark.

[0234] In this way, the summary extraction operator of this embodiment can perform batch summary extraction processing on multiple lines of text data in the target text data to obtain a text summary for each line of text data.

[0235] In an embodiment where the text processing operator is a text similarity analysis operator, in response to the operation of executing the target data processing flowchart, executing the text processing operator according to the target data processing flowchart to process the target text data to obtain the text processing result may include executing the summary extraction operator to perform the following processing steps S3460 to S3470 on the target text data:

[0236] Step S3460 : Encode the first text data and the second text data located in the same row of the target text data respectively to obtain a first vector of the first text data and a second vector of the second text data.

[0237] The first text data is located in a first target column in the target text data, and the second text data is located in a second target column in the target text data.

[0238] In one embodiment of the present disclosure, the target text data may be structured data in multiple columns. The method may further include:

[0239] In response to a request for configuring a processing object for a text similarity analysis operator, a second configuration interface of the text similarity analysis operator is provided; the first target column and the second target column are obtained through the second configuration interface, so that the text similarity analysis operator performs similarity analysis processing on the text data of the first target column and the text data of the second target column in the target text data.

[0240] Specifically, the user may fill in the serial number corresponding to the first target column and the serial number corresponding to the second target column through the second configuration interface.

[0241] The electronic device obtains the serial number input by the user through the second configuration interface to obtain the first target column and the second target column corresponding to the serial number, and then enables the text similarity analysis operator to perform similarity analysis on the text data of the first target column and the text data of the second target column in the target text data.

[0242] The target text data in this embodiment may include M rows. Then, the M rows of data may be traversed, and the text data of the first target column and the second target column in each row are respectively used as the first text data and the second text data.

[0243] For first text data and second text data located in the same row, the first text data may be encoded according to an embedding model to obtain a first vector, and the second text data may be encoded to obtain a second vector.

[0244] Step S3470 : Determine the predicted similarity between the first text data and the second text data based on the first vector and the second vector as a text processing result.

[0245] In one embodiment of the present disclosure, determining the predicted similarity between the first text data and the second text data based on the first vector and the second vector may include steps S3471 to S3477 as shown below:

[0246] Step S3471: Determine the cosine similarity between the first vector and the second vector.

[0247] For example, the first vector of the first text data can be represented as v1, and the second vector of the second text data can be represented as v2. Then, the cosine similarity cos(v1, v2) between the first vector and the second vector can be determined as follows:

[0248]

[0249] Step S3472: Determine the maximum value of the squares of the first vector and the second vector.

[0250] Specifically, the square of the first vector and the square of the second vector are calculated respectively, and the larger one is determined as the maximum value of the squares of the first vector and the second vector, which can be expressed as max(v1,v2) 2 .

[0251] Step S3473: Determine the absolute value of the difference between the first vector and the second vector.

[0252] The absolute value of the difference between the first vector and the second vector can be expressed as |v1-v2|.

[0253] Step S3474: determine the dot product of the first vector and the second vector.

[0254] Step S3475 , concatenating the cosine similarity, the maximum square value, the absolute value of the difference, the dot product, the first vector and the second vector to obtain a target concatenated vector of the first text data and the second text data.

[0255] Specifically, the cosine similarity, the maximum value of the square, the absolute value of the difference, the dot product, the first vector and the second vector may be concatenated in a preset order to obtain a target concatenated vector of the first text data and the second text data.

[0256] The preset order may be set in advance according to application scenarios or specific requirements.

[0257] Step S3476: Obtain a mapping function between the splicing vector and the similarity.

[0258] In one embodiment of the present disclosure, the mapping function may be pre-trained based on training samples, wherein each training sample may include the value of the concatenation vector of two corresponding text data and the pre-labeled similarity between the two text data.

[0259] Specifically, the step of training the mapping function according to the training sample may include the following steps S3476-1 to S3476-:

[0260] Step S3476-1, according to the value of the splicing vector of each training sample, with the undetermined parameters of the preset machine learning algorithm as variables, determine the predicted similarity expression of each training sample.

[0261] For example, for the mth training sample x m , the predicted similarity expression of the training sample can be expressed as F(x m ).

[0262] Step S3476-2: Construct a loss function based on the predicted similarity expression and the corresponding annotation similarity of each training sample.

[0263] In one embodiment of the present disclosure, a cross entropy loss function may be used to determine the predicted similarity expression of the training sample and the labeled similarity.

[0264] In this embodiment, engineers may determine in advance based on experience whether the two texts corresponding to each training sample are similar, and mark the corresponding annotation similarity as 0 or 1. For example, if the two texts corresponding to the training sample are similar, the corresponding annotation similarity may be marked as 1, and if the two texts corresponding to the training sample are not similar, the corresponding annotation similarity may be marked as 0.

[0265] The prediction similarity expression corresponding to the mth training sample is pm , the annotation similarity corresponding to the mth training sample is expressed as y m , when the number of training samples is Z, the loss function L can be expressed as:

[0266]

[0267] Step S3476-3, determine the pending parameters based on the loss function to complete the training of the mapping function.

[0268] In one embodiment of the present disclosure, the mapping function may be obtained by determining the value of the undetermined parameter when the loss function is minimized.

[0269] Step S3477: Determine the predicted similarity between the first text data and the second text data according to the mapping function and the target concatenation vector.

[0270] Specifically, the target splicing vector obtained in step S3475 may be input into a mapping function to obtain the similarity corresponding to the target splicing vector, ie, the predicted similarity between the first text data and the second text data.

[0271] Through the text similarity analysis operator of this embodiment, similarity analysis can be performed on the first text data and the second text data located in the same row in the first target column and the second target column in the target text data in batches to obtain the predicted similarity between the first text data and the second text data in each row.

[0272] Based on any of the above embodiments, the method may further include: displaying the text processing result for the user to view.

[0273] <Example 3>

[0274] Based on the above-mentioned embodiment 1 or embodiment 2, this embodiment provides a method for processing text data based on a text processing operator in a machine learning platform, which may specifically include the following: Figure 5 Steps S5100 to S5200 are shown:

[0275] Step S5100 , in response to the request to bring the text processing operator online, the text processing operator is packaged and processed to obtain a target estimation service for use by the target user.

[0276] In one embodiment of the present disclosure, in response to a request to bring a text processing operator online, packaging the text processing operator to obtain a target estimation service may include the following steps S5110 to S5140:

[0277] Step S5110: In response to a request to create an estimation service, at least one estimation module is provided for user selection.

[0278] In one embodiment of the present disclosure, a button for triggering a request to create an estimation service may be provided, and a user may trigger the request to create an estimation service by clicking the button.

[0279] The estimation module provided in this embodiment can be pre-set according to the application scenario or specific requirements. For example, at least one estimation module may include: a self-learning module, a batch estimation module, a GDBT real-time estimation module, a TensorFlow real-time estimation module, an H2O real-time estimation module, a PMML real-time estimation module, a custom operator estimation module, and a custom application module. Figure 7 shown.

[0280] Step S5120: Obtain the target estimation module selected by the user and provide a model selection interface.

[0281] Users can target Figure 7 Any one of the estimation modules shown performs a selection operation, and the electronic device provides a model selection interface in response to the operation.

[0282] In one embodiment, the selection operation for the target estimation module may be an operation of double-clicking the target estimation module, or an operation of right-clicking the target estimation module and clicking a selection button in a pop-up menu (including a selection button).

[0283] In one embodiment of the present disclosure, the model selection interface may be as follows: Figure 8 shown.

[0284] Step S5130: Obtain the text processing operator selected by the user through the model selection interface.

[0285] In such Figure 8 In the example shown, the user can input search conditions through the input box, and the electronic device can display model operators matching the search conditions for the user to select.

[0286] Step S5140: According to the target estimation module, the text processing operators are packaged and processed to obtain the target estimation service.

[0287] Step S5200: Run the target estimation service, process the text data provided by the target user to the target estimation service, and return the corresponding text processing result to the target user.

[0288] In one embodiment of the present disclosure, the target user may provide text data acquired in real time to the target estimation service, so that the target estimation service processes the text data provided by the target user and returns the corresponding text processing result to the target user.

[0289] In the case where the text processing operator is a summary extraction operator, the way in which the target estimation service performs summary extraction processing on the text data provided by the target user can refer to the way in which the summary extraction operator performs summary extraction processing on the target text data in the aforementioned embodiment 2, and will not be repeated here.

[0290] In the case where the text processing operator is a text similarity analysis operator, the way in which the target estimation service performs similarity analysis on the text data provided by the target user can refer to the way in which the text similarity analysis operator performs similarity analysis on the target text data in the aforementioned embodiment 2, and will not be repeated here.

[0291] Through this embodiment, the text processing operator is packaged into a target estimation service, and the target estimation service is used to process the text data provided by the target user in real time, and the corresponding text processing results are returned to the target user. In this way, text processing can be performed on the real-time text data provided by the target user.

[0292] In one embodiment of the present disclosure, the method may further include:

[0293] In response to a request to configure the target estimation service, a third configuration interface is provided; configuration information of the target estimation service is obtained through the third configuration interface, so as to run the target estimation service according to the configuration information.

[0294] In one embodiment of the present disclosure, the configuration information may include one of the following: GPU resources used, CPU resources used, memory resources used, and the address of the image called at runtime.

[0295] In one embodiment of the present disclosure, the method may further include: displaying the status of the target estimation service in response to a request to view the status of the target estimation service.

[0296] Specifically, a button for the user to trigger a request to view the status of the target estimation service may be provided, and the user triggers the request to view the status of the target estimation service by clicking the button.

[0297] In one embodiment of the present disclosure, the status of the target estimation service may include at least one of the following: resources occupied by the target estimation service during operation, the online history of the target estimation service, the version number of the target estimation service, the deployment time of the target estimation service, and the running time of the target estimation service.

[0298] <Device Example>

[0299] In this embodiment, a text data processing device 5000 is provided. Figure 9As shown, it includes an operator adding module 5100 and a text processing module 5200. The operator adding module 5100 is used to add text processing operators for processing text data to the operator set of the machine learning platform. The text processing operators include: summary extraction operators and / or text similarity analysis operators; the text processing module 5200 is used to process text data based on the text processing operators in the machine learning platform.

[0300] In one embodiment of the present disclosure, the text processing module 5200 may also be used to:

[0301] Provide a user configuration interface, which includes an operator display area and a canvas area for creating a data processing flow chart; wherein the operator display area displays text processing operators for processing text data;

[0302] Get the uploaded target text data;

[0303] In response to the operation of creating a target data processing flowchart, creating a target data processing flowchart in a canvas area according to the target text data and the text processing operator;

[0304] In response to the operation of running the target data processing flow chart, the text processing operator is run according to the target data processing flow chart to process the target text data and obtain a text processing result.

[0305] In one embodiment of the present disclosure, in response to the operation of creating a target data processing flowchart, creating the target data processing flowchart in the canvas area according to the target text data and the text processing operator includes:

[0306] In response to an operation of selecting target text data, displaying the target text data in the canvas area;

[0307] In response to an operation of selecting a text processing operator in the operator display area, displaying the text processing operator in the canvas area;

[0308] In response to the operation of connecting the target text data and the text processing operator, the target text data and the text processing operator are connected in the canvas area to obtain a target data processing flow chart.

[0309] In one embodiment of the present disclosure, the text data processing apparatus 5000 may further include:

[0310] A module for providing a first configuration interface for a text processing operator in response to a request for resource configuration for the text processing operator;

[0311] A module for obtaining resource configuration parameters of a text processing operator through a first configuration interface to run the text processing operator according to the resource configuration parameters.

[0312] In one embodiment of the present disclosure, obtaining the uploaded target text data includes:

[0313] Provide an entry for uploading data;

[0314] In response to an operation of uploading target text data, the target text data uploaded through the portal is acquired.

[0315] In one embodiment of the present disclosure, the text processing operator is a summary extraction operator for performing summary extraction processing on text data;

[0316] Run the text processing operator according to the target data processing flow chart to process the target text data. The text processing results include:

[0317] Run the summary extraction operator to process the target text data as follows:

[0318] Segmenting the target line text data in the target text data to obtain multiple sentences; wherein the target line text data is the text data of any line in the target text data;

[0319] Determine the similarity between each two sentences, and perform regularization on the similarity between each two sentences to obtain a similarity matrix;

[0320] Obtain scores of other sentences based on the similarity matrix and the preset scores of the specified sentence; wherein the other sentences are sentences other than the specified sentence in the multiple sentences;

[0321] According to the scores, a set number of sentences are selected from the multiple sentences as summary sentences;

[0322] Based on the summary sentence, a text summary corresponding to the target row text data is generated as the text processing result.

[0323] In one embodiment of the present disclosure, determining the similarity between each two sentences includes:

[0324] Encode each sentence to obtain the sentence vector of each sentence;

[0325] For every two sentences, the cosine value between the corresponding sentence vectors is determined as the similarity between the two sentences.

[0326] In one embodiment of the present disclosure, a text summary corresponding to the target row of text data is generated based on the summary sentence, and the text processing result includes:

[0327] Get the order of each summary sentence in the target row text data;

[0328] The summary sentences are sorted in order, and preset punctuation is added after each summary sentence to obtain the text summary corresponding to the target row text data as the text processing result.

[0329] In one embodiment of the present disclosure, the text processing operator is a text similarity analysis operator for performing similarity analysis on text data;

[0330] Run the text processing operator according to the target data processing flow chart to process the target file data. The text processing results include:

[0331] Run the text similarity analysis operator to process the target text data as follows:

[0332] Encoding first text data and second text data located in the same row of the target text data respectively to obtain a first vector of the first text data and a second vector of the second text data; wherein the first text data is located in a first target column of the target text data, and the second text data is located in a second target column of the target text data;

[0333] According to the first vector and the second vector, a predicted similarity between the first text data and the second text data is determined as a text processing result.

[0334] In one embodiment of the present disclosure, determining the predicted similarity between the first text data and the second text data according to the first vector and the second vector includes:

[0335] determining a cosine similarity between the first vector and the second vector;

[0336] determining a maximum value of the squares of the first vector and the second vector;

[0337] determining an absolute value of a difference between the first vector and the second vector;

[0338] determining a dot product of a first vector and a second vector;

[0339] Splicing the cosine similarity, the maximum value of the square, the absolute value of the difference, the dot product, the first vector and the second vector to obtain a target splicing vector of the first text data and the second text data;

[0340] Obtain the mapping function between the splicing vector and the similarity;

[0341] Determine a predicted similarity between the first text data and the second text data according to the mapping function and the target concatenation vector.

[0342] In one embodiment of the present disclosure, the text data processing apparatus 5000 may further include:

[0343] A module for providing a second configuration interface of the text similarity analysis operator in response to a request for configuring a processing object of the text similarity analysis operator;

[0344] A module for obtaining a first target column and a second target column through a second configuration interface, so that a text similarity analysis operator performs similarity analysis on text data of the first target column and text data of the second target column in target text data.

[0345] In one embodiment of the present disclosure, the text data processing apparatus 5000 may further include:

[0346] A module for displaying text processing results.

[0347] In one embodiment of the present disclosure, the text processing module 5200 may also be used to:

[0348] In response to a request to bring a text processing operator online, the text processing operator is packaged and processed to obtain a target estimation service for use by the target user;

[0349] Run the target estimation service, process the text data provided by the target user to the target estimation service, and return the corresponding text processing results to the target user.

[0350] In one embodiment of the present disclosure, in response to a request to bring a text processing operator online, the text processing operator is packaged and processed to obtain a target estimation service, including:

[0351] In response to a request to create an estimation service, providing at least one estimation module for user selection;

[0352] Obtain the target estimation module selected by the user and provide a model selection interface;

[0353] Get the text processing operator selected by the user through the model selection interface;

[0354] According to the target estimation module, the text processing operators are packaged and processed to obtain the target estimation service.

[0355] In one embodiment of the present disclosure, the text data processing apparatus 5000 may further include:

[0356] A module for providing a third configuration interface in response to a request to configure the target estimation service;

[0357] A module used to obtain configuration information of the target estimation service through a third configuration interface, so as to run the target estimation service according to the configuration information.

[0358] In one embodiment of the present disclosure, the configuration information includes one of the following: GPU resources used, CPU resources used, memory resources used, and the address of the image called at runtime.

[0359] In one embodiment of the present disclosure, the text data processing apparatus 5000 may further include:

[0360] A module for displaying the status of a target estimation service in response to a request to view the status of the target estimation service.

[0361] In one embodiment of the present disclosure, the status of the target estimation service includes at least one of the following: resources occupied by the target estimation service during operation, the online history of the target estimation service, the version number of the target estimation service, the deployment time of the target estimation service, and the running time of the target estimation service.

[0362] Those skilled in the art will appreciate that the text data processing apparatus 5000 can be implemented in various ways. For example, the text data processing apparatus 5000 can be implemented by configuring a processor with instructions. For example, the instructions can be stored in a ROM, and when the device is started, the instructions are read from the ROM into a programmable device to implement the text data processing apparatus 5000. For example, the text data processing apparatus 5000 can be embedded in a dedicated device (e.g., an ASIC). The text data processing apparatus 5000 can be divided into independent units, or they can be combined together for implementation. The text data processing apparatus 5000 can be implemented by one of the various implementations described above, or by a combination of two or more of the various implementations described above.

[0363] In this embodiment, the text data processing device 5000 can have multiple implementation forms. For example, the text data processing device 5000 can be a functional module running in any software product or application that provides text processing services, or it can be an external embedded component, plug-in, patch, etc. of these software products or applications, or it can also be these software products or applications themselves.

[0364] <System Example>

[0365] In this embodiment, if Figure 10 As shown, a system 6000 is also provided, comprising at least one computing device 6100 and at least one storage device 6200. The at least one storage device 6200 is used to store executable instructions; the instructions are used to control the at least one computing device 6100 to execute the method according to any embodiment of the present disclosure.

[0366] In this embodiment, the system 6000 can be a device such as a mobile phone, a tablet computer, a PDA, a desktop computer, a laptop computer, a workstation, a game console, etc., or it can be a distributed system composed of multiple devices.

[0367] <Computer-readable storage medium>

[0368] In this embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the method of any embodiment of the present disclosure is implemented.

[0369] The present invention may be a device, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0370] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0371] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0372] The computer program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), is personalized by utilizing the state information of the computer readable program instructions, and the electronic circuit can execute the computer readable program instructions, thereby realizing various aspects of the present invention.

[0373] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0374] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0375] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0376] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0377] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A method for processing text data, comprising: Adding a text processing operator for processing text data to the operator set of the machine learning platform, wherein the text processing operator includes: a summary extraction operator and / or a text similarity analysis operator; Performing text data processing based on the text processing operator in the machine learning platform, wherein the machine learning platform can directly call the text processing operator selected by the user to perform corresponding processing on the text data; The processing of text data based on the text processing operator in the machine learning platform includes: Providing a user configuration interface, the user configuration interface including an operator display area and a canvas area for creating a data processing flowchart; wherein the operator display area displays the text processing operator for processing text data, and the data processing flowchart can represent the corresponding data processing process, and the data processing flowchart is a directed acyclic graph; Get the uploaded target text data; In response to an operation of creating a target data processing flowchart, creating the target data processing flowchart in the canvas area according to the target text data and the text processing operator; In response to the operation of running the target data processing flowchart, the text processing operator is run according to the target data processing flowchart to process the target text data and obtain a text processing result.

2. The method according to claim 1, wherein in response to the operation of creating a target data processing flowchart, creating the target data processing flowchart in the canvas area according to the target text data and the text processing operator comprises: In response to an operation of selecting the target text data, displaying the target text data in the canvas area; In response to an operation of selecting the text processing operator in the operator display area, displaying the text processing operator in the canvas area; In response to the operation of connecting the target text data and the text processing operator, the target text data and the text processing operator are connected in the canvas area to obtain the target data processing flow chart.

3. The method according to claim 1, further comprising: In response to a request for resource configuration of the text processing operator, providing a first configuration interface of the text processing operator; The resource configuration parameters of the text processing operator are obtained through the first configuration interface to run the text processing operator according to the resource configuration parameters.

4. The method according to claim 1, wherein obtaining the uploaded target text data comprises: Provide an entry for uploading data; In response to an operation of uploading target text data, the target text data uploaded through the portal is acquired.

5. The method according to claim 1, wherein the text processing operator is a summary extraction operator for performing summary extraction processing on text data; The step of running the text processing operator to process the target text data according to the target data processing flow chart to obtain a text processing result includes: Run the summary extraction operator to process the target text data as follows: Performing sentence segmentation processing on target line text data in the target text data to obtain multiple sentences; wherein the target line text data is text data of any line in the target text data; Determine the similarity between each two sentences, and perform regularization on the similarity between each two sentences to obtain a similarity matrix; Obtaining scores of other sentences based on the similarity matrix and the preset scores of the specified sentence; wherein the other sentences are sentences other than the specified sentence in the plurality of sentences; selecting a set number of sentences from the plurality of sentences as summary sentences based on the scores; According to the summary sentence, a text summary corresponding to the target row of text data is generated as the text processing result.

6. The method according to claim 5, wherein determining the similarity between each two sentences comprises: Encode each sentence to obtain the sentence vector of each sentence; For every two sentences, the cosine value between the corresponding sentence vectors is determined as the similarity between the two sentences.

7. The method according to claim 5, wherein generating a text summary corresponding to the target row of text data as the text processing result based on the summary sentence comprises: Obtaining the order of each summary sentence in the target row of text data; The summary sentences are sorted in the order, and preset punctuation is added after each summary sentence to obtain a text summary corresponding to the target row of text data as the text processing result.

8. The method according to claim 1, wherein the text processing operator is a text similarity analysis operator for performing similarity analysis on text data; The step of running the text processing operator to process the target file data according to the target data processing flow chart to obtain a text processing result includes: Run the text similarity analysis operator to process the target text data as follows: Encoding first text data and second text data located in the same row of the target text data respectively to obtain a first vector of the first text data and a second vector of the second text data; wherein the first text data is located in a first target column of the target text data, and the second text data is located in a second target column of the target text data; According to the first vector and the second vector, a predicted similarity between the first text data and the second text data is determined as the text processing result.

9. The method according to claim 8, wherein determining the predicted similarity between the first text data and the second text data based on the first vector and the second vector comprises: determining a cosine similarity between the first vector and the second vector; determining a maximum of the squares of the first vector and the second vector; determining an absolute value of a difference between the first vector and the second vector; determining a dot product of the first vector and the second vector; Splicing the cosine similarity, the maximum value of the square, the absolute value of the difference, the dot product, the first vector, and the second vector to obtain a target splicing vector of the first text data and the second text data; Obtain the mapping function between the splicing vector and the similarity; Determine a predicted similarity between the first text data and the second text data according to the mapping function and the target concatenation vector.

10. The method according to claim 8, further comprising: In response to a request for configuring a processing object for the text similarity analysis operator, providing a second configuration interface for the text similarity analysis operator; The first target column and the second target column are obtained through the second configuration interface, so that the text similarity analysis operator performs similarity analysis on the text data of the first target column and the text data of the second target column in the target text data.

11. The method according to claim 1 , further comprising: The text processing result is displayed.

12. The method according to claim 1, wherein The processing of text data based on the text processing operator in the machine learning platform includes: In response to a request to bring the text processing operator online, the text processing operator is packaged and processed to obtain a target estimation service for use by a target user; The target estimation service is run to process text data provided by the target user to the target estimation service, and a corresponding text processing result is returned to the target user.

13. The method according to claim 12, wherein, in response to the request to bring the text processing operator online, the step of packaging the text processing operator to obtain the target estimation service comprises: In response to a request to create an estimation service, providing at least one estimation module for user selection; Obtain the target estimation module selected by the user and provide a model selection interface; Obtaining the text processing operator selected by the user through the model selection interface; According to the target estimation module, the text processing operators are packaged to obtain the target estimation service.

14. The method according to claim 12, further comprising: In response to a request to configure the target estimation service, providing a third configuration interface; The configuration information of the target estimation service is obtained through the third configuration interface, so as to run the target estimation service according to the configuration information.

15. The method according to claim 14, wherein the configuration information includes one of the following: GPU resources used, CPU resources used, memory resources used, and the address of the image called during runtime.

16. The method according to claim 12, further comprising: In response to a request to view the status of the target estimation service, the status of the target estimation service is displayed.

17. According to the method of claim 16, the status of the target estimation service includes at least one of the following: resources occupied by the target estimation service during operation, the online history of the target estimation service, the version number of the target estimation service, the deployment time of the target estimation service, and the running time of the target estimation service.

18. A text data processing device, comprising: An operator adding module, configured to add a text processing operator for processing text data to the operator set of the machine learning platform, wherein the text processing operator includes: a summary extraction operator and / or a text similarity analysis operator; A text processing module, configured to process text data based on the text processing operator in the machine learning platform, wherein the machine learning platform can directly call the text processing operator selected by the user to perform corresponding processing on the text data; The text processing module is also used for: Providing a user configuration interface, the user configuration interface including an operator display area and a canvas area for creating a data processing flowchart; wherein the operator display area displays the text processing operator for processing text data, and the data processing flowchart can represent the corresponding data processing process, and the data processing flowchart is a directed acyclic graph; Get the uploaded target text data; In response to an operation of creating a target data processing flowchart, creating the target data processing flowchart in the canvas area according to the target text data and the text processing operator; In response to the operation of running the target data processing flowchart, the text processing operator is run according to the target data processing flowchart to process the target text data and obtain a text processing result.

19. The apparatus according to claim 18, wherein in response to the operation of creating a target data processing flowchart, creating the target data processing flowchart in the canvas area according to the target text data and the text processing operator comprises: In response to an operation of selecting the target text data, displaying the target text data in the canvas area; In response to an operation of selecting the text processing operator in the operator display area, displaying the text processing operator in the canvas area; In response to the operation of connecting the target text data and the text processing operator, the target text data and the text processing operator are connected in the canvas area to obtain the target data processing flow chart.

20. The apparatus of claim 18, further comprising: A module for providing a first configuration interface for the text processing operator in response to a request for resource configuration for the text processing operator; A module for obtaining resource configuration parameters of the text processing operator through the first configuration interface to run the text processing operator according to the resource configuration parameters.

21. The apparatus according to claim 18, wherein the acquiring the uploaded target text data comprises: Provide an entry for uploading data; In response to an operation of uploading target text data, the target text data uploaded through the portal is acquired.

22. The apparatus according to claim 18, wherein the text processing operator is a summary extraction operator for performing summary extraction processing on text data; The step of running the text processing operator to process the target text data according to the target data processing flow chart to obtain a text processing result includes: Run the summary extraction operator to process the target text data as follows: Performing sentence segmentation processing on target line text data in the target text data to obtain multiple sentences; wherein the target line text data is text data of any line in the target text data; Determine the similarity between each two sentences, and perform regularization on the similarity between each two sentences to obtain a similarity matrix; Obtaining scores of other sentences based on the similarity matrix and the preset scores of the specified sentence; wherein the other sentences are sentences other than the specified sentence in the plurality of sentences; selecting a set number of sentences from the plurality of sentences as summary sentences based on the scores; According to the summary sentence, a text summary corresponding to the target row of text data is generated as the text processing result.

23. The apparatus according to claim 22, wherein determining the similarity between each two sentences comprises: Encode each sentence to obtain the sentence vector of each sentence; For every two sentences, the cosine value between the corresponding sentence vectors is determined as the similarity between the two sentences.

24. The apparatus according to claim 22, wherein generating a text summary corresponding to the target row of text data based on the summary sentence as the text processing result comprises: Obtaining the order of each summary sentence in the target row of text data; The summary sentences are sorted in the order, and preset punctuation is added after each summary sentence to obtain a text summary corresponding to the target row of text data as the text processing result.

25. The apparatus according to claim 18, wherein the text processing operator is a text similarity analysis operator for performing similarity analysis on text data; The step of running the text processing operator to process the target file data according to the target data processing flow chart to obtain a text processing result includes: Run the text similarity analysis operator to process the target text data as follows: Encoding first text data and second text data located in the same row of the target text data respectively to obtain a first vector of the first text data and a second vector of the second text data; wherein the first text data is located in a first target column of the target text data, and the second text data is located in a second target column of the target text data; According to the first vector and the second vector, a predicted similarity between the first text data and the second text data is determined as the text processing result.

26. The apparatus according to claim 25, wherein determining the predicted similarity between the first text data and the second text data based on the first vector and the second vector comprises: determining a cosine similarity between the first vector and the second vector; determining a maximum of the squares of the first vector and the second vector; determining an absolute value of a difference between the first vector and the second vector; determining a dot product of the first vector and the second vector; Splicing the cosine similarity, the maximum value of the square, the absolute value of the difference, the dot product, the first vector, and the second vector to obtain a target splicing vector of the first text data and the second text data; Obtain the mapping function between the splicing vector and the similarity; Determine a predicted similarity between the first text data and the second text data according to the mapping function and the target concatenation vector.

27. The apparatus according to claim 25, further comprising: A module for providing a second configuration interface of the text similarity analysis operator in response to a request for configuring a processing object of the text similarity analysis operator; A module for obtaining the first target column and the second target column through the second configuration interface, so that the text similarity analysis operator performs similarity analysis on the text data of the first target column and the text data of the second target column in the target text data.

28. The apparatus of claim 18, further comprising: A module for displaying the text processing results.

29. The apparatus according to claim 18, wherein The text processing module is also used for: In response to a request to bring the text processing operator online, the text processing operator is packaged and processed to obtain a target estimation service for use by a target user; The target estimation service is run to process text data provided by the target user to the target estimation service, and a corresponding text processing result is returned to the target user.

30. The apparatus according to claim 29, wherein, in response to the request to bring the text processing operator online, the step of packaging the text processing operator to obtain a target estimation service comprises: In response to a request to create an estimation service, providing at least one estimation module for user selection; Obtain the target estimation module selected by the user and provide a model selection interface; Obtaining the text processing operator selected by the user through the model selection interface; According to the target estimation module, the text processing operators are packaged to obtain the target estimation service.

31. The apparatus of claim 29, further comprising: A module for providing a third configuration interface in response to a request to configure the target estimation service; A module for obtaining configuration information of the target estimation service through the third configuration interface, so as to run the target estimation service according to the configuration information.

32. The device according to claim 31, wherein the configuration information includes one of the following: GPU resources used, CPU resources used, memory resources used, and the address of the image called during runtime.

33. The apparatus of claim 29, further comprising: Means for displaying a status of the target estimation service in response to a request to view the status of the target estimation service.

34. The device according to claim 33, wherein the status of the target estimation service includes at least one of the following: resources occupied by the target estimation service during operation, the online history of the target estimation service, the version number of the target estimation service, the deployment time of the target estimation service, and the running time of the target estimation service.

35. A system comprising at least one computing device and at least one storage device, wherein: The at least one storage device is used to store instructions, and the instructions are used to control the at least one computing device to execute the method according to any one of claims 1 to 17.

36. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 17 when executed by a processor.

Images