Data processing method, device and electronic equipment

Abstract

The present disclosure relates to a method, device and electronic device for data processing, and to the field of electronic information technology. The method includes: for each first text data in a plurality of first text data, determining first description information of the first text data in a first preset analysis dimension; determining abnormal text data and abnormal analysis dimensions corresponding to the abnormal text data from the plurality of first text data according to the first description information of each first text data; the abnormal analysis dimension refers to at least one of the first preset analysis dimensions; determining data analysis requirements according to the abnormal analysis dimensions, and performing data analysis on the second text data according to the data analysis requirements. In this way, the downstream model can perform data analysis on the second text data to be analyzed based on the data analysis requirements, whether before, during or after training, thereby ensuring the reliability and credibility of the data analysis results.

Description

Technical Field

[0001] The present disclosure relates to the field of electronic information technology, and in particular, to a data processing method, device and electronic equipment. Background Art

[0002] With the rapid development of artificial intelligence, artificial intelligence technology has been widely used in many fields. Natural language processing is an important aspect of artificial intelligence technology. It allows computers to understand and process human language. The fields involved in natural language processing are gradually extending to tasks such as word segmentation, semantic classification and matching, and question answering.

[0003] At present, how to automate the data analysis and generation of natural language processing models before, during, and after training is an urgent problem to be solved. Summary of the invention

[0004] The purpose of the present disclosure is to provide a method, device and electronic device for data processing, which mines abnormal text data and uses the abnormal text data as an analysis basis for data analysis to obtain corresponding data analysis requirements, so as to realize automated data analysis of second text data to be analyzed.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for data processing is provided, the method comprising: For each first text data among the plurality of first text data, determining first description information of the first text data in a first preset analysis dimension; According to the first description information of each first text data, determine abnormal text data and abnormal analysis dimensions corresponding to the abnormal text data from the plurality of first text data; the abnormal analysis dimension refers to at least one of the first preset analysis dimensions; Data analysis requirements are determined according to the abnormal analysis dimension, and data analysis is performed on the second text data according to the data analysis requirements.

[0006] According to a second aspect of an embodiment of the present disclosure, a data processing device is provided, the device comprising: An acquisition module is configured to determine, for each first text data among a plurality of first text data, first description information of the first text data in a first preset analysis dimension; A determination module is configured to determine abnormal text data and abnormal analysis dimensions corresponding to the abnormal text data from the plurality of first text data according to the first description information of each first text data; the abnormal analysis dimension refers to at least one of the first preset analysis dimensions; The analysis module is configured to determine data analysis requirements according to the abnormal analysis dimension, and perform data analysis on the second text data according to the data analysis requirements.

[0007] According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data processing method provided by the first aspect of the present disclosure are implemented.

[0008] According to a fourth aspect of an embodiment of the present disclosure, there is provided an electronic device, including: a memory having a computer program stored thereon; A processor is used to execute the computer program in the memory to implement the steps of the data processing method provided in the first aspect of the present disclosure.

[0009] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the data processing method provided in the first aspect of the present disclosure.

[0010] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects: First, for each of the first text data in the plurality of first text data, the first description information of the first text data in the first preset analysis dimension is determined. Secondly, according to the first description information of each of the first text data, the abnormal text data and the abnormal analysis dimension corresponding to the abnormal text data are determined from the plurality of first text data; the abnormal analysis dimension refers to at least one of the first preset analysis dimensions. Then, the data analysis requirement is determined according to the abnormal analysis dimension, and the second text data is analyzed according to the data analysis requirement. Through the above scheme, based on the first description information corresponding to each of the first text data in the first preset analysis dimension, the plurality of first text data are detected, and the abnormal text data in the data and the abnormal analysis dimension corresponding to the abnormal text data are automatically mined. And the abnormal analysis dimension is used as the analysis breakthrough of data analysis to obtain the corresponding data analysis requirement, enriching the idea of ​​data analysis mining. Then, according to the data analysis requirement, the second text data to be analyzed is analyzed. In this way, the downstream model can perform data analysis on the second text data to be analyzed based on the data analysis requirement, whether before training, during training, or after training, to ensure the reliability and credibility of the data analysis results. At the same time, it also provides support for the training optimization of the subsequent downstream model.

[0011] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings: Figure 1 It is a schematic diagram of an application environment according to an exemplary embodiment.

[0013] Figure 2 The present invention is a flowchart of a data processing method according to an exemplary embodiment.

[0014] Figure 3 The present invention is a flowchart of another data processing method according to an exemplary embodiment.

[0015] Figure 4 The invention is a block diagram of a data processing device according to an exemplary embodiment.

[0016] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0017] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0018] The terms "first", "second", etc. in the specification and claims of this application and the above drawings are used to distinguish similar objects and do not have to be understood as a specific order or sequence. In addition, in the description with reference to the drawings, the same symbols in different drawings represent the same elements.

[0019] In the description of the present disclosure, unless otherwise specified, "multiple" means two or more than two, and other quantifiers are similar thereto; "at least one item", "one item or multiple items" or similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one item a can represent any number of a; for another example, one item or multiple items among a, b and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple; "and / or" is a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " indicates that the associated objects before and after are in an "or" relationship.

[0020] Although operations or steps are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood that it is required to perform these operations or steps in the specific order shown or in a serial order, or to perform all the operations or steps shown to obtain the desired results. In the embodiments of the present disclosure, these operations or steps can be performed in series; these operations or steps can also be performed in parallel; or some of these operations or steps can be performed.

[0021] The following is a brief description of the application scenario of the solution provided in the embodiment of the present application. The present disclosure can be applied in scenarios where data analysis is required, such as data analysis of models before, during, and after training. With the rapid development of artificial intelligence, artificial intelligence technology has been widely used in many fields. Natural language processing (English: Natural Language Processing; referred to as: NLP) is an important aspect of artificial intelligence technology. It allows computers to understand and process human language. The fields involved in natural language processing are gradually extended to tasks such as word segmentation, semantic classification and matching, and question and answer. The data analysis of natural language processing models before, during, and after training has become a key link in optimizing the model. Before each model is officially enabled, the badcase nodes (i.e., the sample cases where the inference results obtained by the model in reasoning are inconsistent with the labels) after the formal activation need to analyze the data. At present, for different scenarios, it is necessary to manually set the data analysis standard of the scenario, which has poor reusability, low efficiency, and is easily affected by chance, resulting in the analysis dimension not necessarily being comprehensive. Therefore, in this embodiment, this solution can perform data detection on multiple first text data, and automatically mine abnormal text data in the data and abnormal analysis dimensions corresponding to the abnormal text data. The abnormal analysis dimension is used as a breakthrough point for data analysis to obtain the corresponding data analysis requirements, enriching the ideas of data analysis and mining. Then, according to the data analysis requirements, the second text data to be analyzed is analyzed.

[0022] In this way, the downstream model can perform data analysis based on the determined data analysis needs before, during, or after training, ensuring the reliability and credibility of the data analysis results. The entire process does not require human intervention, which improves the efficiency and automation of data analysis. At the same time, it also provides support for the training and optimization of subsequent models with data analysis results, facilitates a more comprehensive understanding of the second text data, and provides data support for later model activation. On the one hand, before the model is activated, more scientific and appropriate algorithm indicators can be formulated based on the data analysis results obtained by the data analysis method provided by this implementation, and the training data and test data of the model can be constructed based on the data analysis results; on the other hand, after the model is activated, the training data of the model can be adjusted based on the data analysis results obtained by the data analysis method provided by this implementation, thereby optimizing the model.

[0023] In order to better understand a data processing method, device, and electronic device provided in an embodiment of the present application, an application environment applicable to the embodiment of the present application is described below.

[0024] See also Figure 1 , Figure 1 A schematic diagram of an application environment of the data processing method provided by an embodiment of the present application is shown. As an implementation mode, the data processing method provided by the embodiment of the present application can be applied to an electronic device. The electronic device can be as follows: Figure 1 The server 110 shown in the figure can be connected to the terminal device 120 through a network. The network is used to provide a medium for a communication link between the server 110 and the terminal device 120. The network can include various connection types, such as a wired communication link, a wireless communication link, etc., which are not limited in the embodiments of the present application. Optionally, in other embodiments, the electronic device can also be a smart phone, a laptop computer, etc.

[0025] It should be understood that Figure 1 The server 110, network and terminal device 120 are merely illustrative. Depending on the implementation requirements, there may be any number of servers, networks and terminal devices. For example, the server 110 may be a physical server or a server cluster composed of multiple servers, and the terminal device 120 may be a mobile phone, a tablet, a desktop computer, a laptop computer, and the like. It is understood that the embodiments of the present application may also allow multiple terminal devices 120 to access the server 110 at the same time.

[0026] In some embodiments, the terminal device 120 can obtain the first description information corresponding to the first preset analysis dimension for each of the first text data. Further, the terminal device 120 sends the first description information corresponding to the first preset analysis dimension for each of the first text data to the server 110 through the network. After the server 110 receives the first description information corresponding to the first preset analysis dimension for each of the first text data, the data can be processed by the data processing method described in the embodiment of the present application.

[0027] The specific implementation modes of the present invention are described in detail below with reference to the accompanying drawings.

[0028] Figure 2 is a flow chart of a data processing method according to an exemplary embodiment. Figure 2 As shown, the method may include the following steps: In step S201, for each first text data among a plurality of first text data, first description information of the first text data in a first preset analysis dimension is determined.

[0029] The first text data is data corresponding to the target scenario, and the target scenario can be a scenario where the target model is applied, such as a dialogue scenario (such as a collection conversation scenario), a question-and-answer scenario, an advertising scenario, etc. The first text data can be a dialogue text or a single sentence text, and the present disclosure does not specifically limit this. The target model can be any model that needs to be trained. Before the target model is trained, the first text data can be unlabeled data; during the target model training process or after it is officially activated, the first text data can be labeled data.

[0030] In some embodiments, the preset analysis dimension can be understood as the features of text data corresponding to different dimensions. For example, the preset analysis dimension can include data source, number of role dialogues, number of conversation rounds, sentence length, conversation duration, etc. The description information is the specific content of the text data corresponding to the preset analysis dimension. For example, when the preset analysis dimension includes the data source, the description information can include speech (Automatic Speech Recognition, ASR) translation text (i.e., text obtained by speech conversion), OCR (Optical Character Recognition, optical character recognition) translation text (i.e., text obtained by scanning recognition), text, etc. For another example, when the preset analysis dimension includes the number of dialogue roles, the description information can include a single role, two-party conversation (e.g., it can include a seat and a customer), a three-party conversation (e.g., it can include seat 1 & seat 2 & customer, or seat & customer 1 & customer 2), etc.

[0031] In this step, the first preset analysis dimension may be a preset analysis dimension corresponding to the preset selected first text data, and the first preset analysis dimension may include one or more. Accordingly, the first description information may be description information corresponding to the first preset analysis dimension.

[0032] In step S202, abnormal text data and abnormal analysis dimensions corresponding to the abnormal text data are determined from the plurality of first text data according to the first description information of each first text data.

[0033] The abnormal analysis dimension refers to at least one of the first preset analysis dimensions.

[0034] In this step, firstly, abnormal text data may be determined from a plurality of first text data according to the first description information; and then, an abnormal analysis dimension corresponding to each abnormal text data may be determined according to the first description information.

[0035] In some embodiments, the abnormal text data can be determined from the plurality of first text data by comparing the first similarities between each two first text data on different first preset analysis dimensions, and the abnormal analysis dimension corresponding to the abnormal text data can be determined from the first preset analysis dimension based on the first similarities between each two first text data.

[0036] It should be noted that the abnormal analysis dimension may refer to the analysis dimension itself, such as the number of conversation rounds, or may refer to the reason for the abnormality of the analysis dimension, such as the lack of data for a certain role.

[0037] In step S203, data analysis requirements are determined according to the abnormal analysis dimension, and data analysis is performed on the second text data according to the data analysis requirements.

[0038] Among them, after determining the abnormal analysis dimension, the intention recognition can be performed on each abnormal analysis dimension to obtain the data analysis requirements corresponding to each abnormal analysis dimension.

[0039] For example, if the abnormal analysis dimension includes the number of conversation rounds, after the intent is identified, the data analysis requirement is to find the number of conversation rounds. If the abnormal analysis dimension includes that the data of a certain role is too small, after the intent is identified, the data analysis requirement is to determine the data ratio of each role.

[0040] After the data analysis requirement is determined, a corresponding data analysis code can be generated according to the data analysis requirement, and the data analysis code is used to analyze the text data to be analyzed. Afterwards, the data analysis code can be used to perform data analysis on the second text data to be analyzed.

[0041] In some embodiments, the target data analysis code may be compiled to obtain a corresponding data analysis program, and then the second text data to be analyzed may be loaded and the data analysis program may be run to implement analysis of the second text data.

[0042] In this way, in the process of downstream application, the second text data to be analyzed can be input into the data analysis program, so as to perform data analysis on the second text data through the data analysis program, as data support in the data application process, and the relevant data of the target model in the training or optimization process can be adjusted through the data analysis results, so as to remove obstacles before and after the target model training, and ensure the smooth progress of the automatic iteration of the target model. It can be seen that in this embodiment, the data analysis process is standardized and streamlined through automation, which greatly improves the efficiency and comprehensiveness of data analysis.

[0043] By adopting the above method, based on the first description information corresponding to the first preset analysis dimension of each first text data, data detection is performed on multiple first text data, and abnormal text data in the data and abnormal analysis dimensions corresponding to the abnormal text data are automatically mined. The abnormal analysis dimension is used as an analytical breakthrough for data analysis to obtain the corresponding data analysis requirements, enriching the idea of ​​data analysis and mining. Then, according to the data analysis requirements, data analysis is performed on the second text data to be analyzed. In this way, the downstream model can perform data analysis on the second text data to be analyzed based on the data analysis requirements before, during, or after training, thereby ensuring the reliability and credibility of the data analysis results. At the same time, it also provides support for the data analysis results for the subsequent training optimization of downstream models.

[0044] The above step 202 is described in detail below. In a possible implementation, Figure 3 As shown, in the above step S202, determining abnormal text data from the multiple first text data according to the first description information of each first text data, and the abnormal analysis dimension corresponding to the abnormal text data may include the following steps: In step S2021 , the first similarity between every two first text data is determined according to the first description information of each first text data.

[0045] In this step, the sentence vector corresponding to each first description information can be determined first. Then, for each first preset analysis dimension, the first similarity between the sentence vectors of the first description information corresponding to each two first text data is determined. Specifically, the first similarity corresponding to each two first text data can be obtained by calculating the vector distance between the sentence vectors of the first description information corresponding to each two first text data.

[0046] In step S2022, abnormal text data is determined from the plurality of first text data according to the first similarity, and an abnormality analysis dimension corresponding to the abnormal text data is determined according to the first similarity.

[0047] In some embodiments, the number of the first preset analysis dimensions is multiple, and determining the abnormal text data from the multiple first text data according to the first similarity in step S2022 may include: Step A1: for each first text data, based on the first similarity corresponding to the first text data under each first preset analysis dimension, determine the sum of similarities corresponding to the first text data.

[0048] In this step, the sum of similarities corresponding to the first text data may be determined in the following two ways.

[0049] Method 1: for each first text data, all first similarities corresponding to the first text data are added together to obtain the sum of the first similarities corresponding to the first text data.

[0050] Method 2: Obtain a dimension weight corresponding to each first preset analysis dimension, and perform a preset operation on the first similarity corresponding to the first text data under each first preset analysis dimension according to the dimension weight to obtain the sum of similarities corresponding to the first text data.

[0051] The preset operation may be, for example, a weighted sum operation. Considering that in actual application scenarios, different first preset analysis dimensions have different importance in determining whether the first text data is abnormal, the dimension weights corresponding to different first preset analysis dimensions may be preset, so that the abnormal text data in the multiple first text data may be determined in combination with the dimension weight corresponding to each first preset analysis dimension, so as to improve the rationality of the abnormal text data determination.

[0052] For example, the dimension weight corresponding to each first preset analysis dimension can be preset, and a larger dimension weight can be set for a more important first preset analysis dimension, and a smaller dimension weight can be set for a less important first preset analysis dimension. In this way, when determining the sum of similarities, all first similarities corresponding to the first text data can be weighted and summed according to the dimension weight of each first preset analysis dimension to obtain the sum of similarities corresponding to the first text data.

[0053] The higher the sum of similarities is, the more similar the first text data is to other first text data; the lower the sum of similarities is, the less similar the first text data is to other first text data.

[0054] Step A2: among the plurality of first text data, first text data whose sum of similarities is less than or equal to a first preset similarity threshold is determined as abnormal text data.

[0055] When the sum of the similarities is less than or equal to the first preset similarity threshold, it can be determined that there are large differences between the current first text data and other first text data in multiple first preset analysis dimensions. Therefore, the first text data can be regarded as abnormal text data.

[0056] In this way, the above-mentioned abnormality judgment can be performed on each first text data to obtain abnormal text data among multiple first text data, and the abnormal text data can be used as the analysis basis for subsequent data analysis, thereby ensuring the comprehensiveness and accuracy of subsequent data analysis. At the same time, by setting the dimension weight corresponding to each first preset analysis dimension, the rationality of subsequent data analysis can be further ensured.

[0057] In some other embodiments, the number of the first preset analysis dimensions is multiple, and in step S2022, determining the abnormal analysis dimension corresponding to the abnormal text data according to the first similarity may include: Step B1: for each first preset analysis dimension, among the first similarities corresponding to the first preset analysis dimension, determine the first similarity related to the abnormal text data as the second similarity.

[0058] In this step, after determining the abnormal text data, it is necessary to further determine the cause of the abnormal text data, that is, the abnormal analysis dimension corresponding to the abnormal text data. Specifically, the determination can be made based on the second similarity corresponding to each abnormal text data, and the second similarity can obtain the first similarity related to the abnormal text data from the multiple first similarities obtained above.

[0059] Step B2: determining the abnormal analysis dimension corresponding to the abnormal text data according to the second similarity of the abnormal text data under each first preset analysis dimension.

[0060] In this step, first, for each first preset analysis dimension, the quantitative proportion of the third similarity in the second similarity of the first preset analysis dimension is determined. The third similarity is the second similarity in the second similarity under the first preset analysis dimension that is less than or equal to the second preset similarity threshold. Then, the first preset analysis dimension whose quantitative proportion is greater than or equal to the preset quantitative proportion is determined as an abnormal analysis dimension.

[0061] For example, if the first text data includes text data A, text data B, text data C and text data D, and text data A is abnormal text data, on the first preset analysis dimension, the second similarity between text data A and text data B is 18%, the second similarity between text data A and text data C is 5%, and the second similarity between text data A and text data D is 80%. The second preset similarity threshold is set to 40%, and the preset number ratio is 60%. It can be determined that the third similarities of the multiple second similarities that are less than or equal to the second preset similarity threshold include 18% and 5%. The number of third similarities corresponding to text data A is 2, the number of multiple second similarities corresponding to text data A is 3, and the number ratio corresponding to text data A is 2 / 3. It can be seen that the number ratio is greater than or equal to the preset number ratio, and the first preset analysis dimension can be determined as the abnormal analysis dimension corresponding to text data A.

[0062] Of course, the abnormal analysis dimension can also be determined by the proportion of the number of similarities with higher similarities. That is, the second similarity greater than the second preset similarity threshold in the second similarity under the first preset analysis dimension is used as the fourth similarity, and the proportion of the number of the fourth similarity in the second similarity of the first preset analysis dimension is determined, and then it is determined whether its proportion is less than the preset proportion. And if it is determined that its proportion is less than the preset proportion, the first preset analysis dimension is used as the abnormal analysis dimension.

[0063] In addition, the abnormality analysis dimension may be determined directly by determining whether the number of the third similarities is greater than or equal to a preset number threshold. For example, when the number of the third similarities is greater than or equal to the preset number threshold, the first preset analysis dimension is used as the abnormality analysis dimension.

[0064] In another possible implementation, determining abnormal text data from the multiple first text data and the abnormal analysis dimensions corresponding to the abnormal text data in the above-mentioned step S202 according to the first description information of each first text data may include: inputting the multiple first text data and the first description information corresponding to each first text data into a pre-trained third model to obtain the abnormal text data output by the third model and the abnormal analysis dimensions corresponding to the abnormal text data.

[0065] Among them, the third model can be an abnormal data detection model, which can be understood as a classification model, which aims to perform comparative detection on multiple dimensions (i.e., first preset analysis dimensions) of multiple first text data, so as to select abnormal text data and abnormal analysis dimensions corresponding to the abnormal text data.

[0066] Specifically, the third model can perform vector conversion on the first description information corresponding to the input multiple first text data to obtain the corresponding sentence vector. And perform similarity calculation on the sentence vector for every two first text data under each first preset analysis dimension, and then determine the abnormal text data and the abnormal analysis dimension corresponding to the abnormal text data from the multiple first text data according to the similarity calculation result. For example, the third model outputs the abnormal text data and the abnormal analysis dimension corresponding to the abnormal text data, which can be output in the form of a list.

[0067] In addition, after obtaining the abnormal text data and the abnormal analysis dimensions corresponding to the abnormal text data, the abnormal text data and the abnormal analysis dimensions corresponding to the abnormal text data can also be recorded, and an abnormal text data analysis report can be generated to facilitate subsequent review by relevant technical personnel.

[0068] The method for determining the data analysis requirement according to the abnormal analysis dimension in the above step S203 is described in detail below.

[0069] In some embodiments, the abnormal analysis dimension may be subjected to intent identification to obtain the data analysis requirement.

[0070] Specifically, the fourth model can be used to perform intent recognition on each abnormal analysis dimension to obtain the data analysis requirements corresponding to each abnormal analysis dimension. The fourth model can be an intent recognition model in the related art.

[0071] In other embodiments, in order to further improve the comprehensiveness and reliability of data analysis, the factors that cause the abnormal text data to be abnormal may be further considered, and the data analysis requirements may be determined in combination with the factors. Specifically, the following steps may be included: (1) According to the anomaly analysis dimension, determine the influencing factors that affect the occurrence of the anomaly in the abnormal text data.

[0072] In a possible implementation, the influencing factor corresponding to the abnormal analysis dimension may be determined according to the correspondence between the abnormal analysis dimension and the preset influencing factor, wherein the preset influencing factor correspondence includes the correspondence between the preset analysis dimension and the preset influencing factor.

[0073] That is, preset influencing factors related to each preset analysis dimension can be preset to obtain a preset influencing factor correspondence, and each preset analysis dimension corresponds to one or more preset influencing factors. Through the preset influencing factor correspondence, the influencing factor corresponding to the abnormal analysis dimension can be obtained.

[0074] For example, when the preset analysis dimension includes the number of conversation rounds, the preset influencing factors corresponding to the preset analysis dimension may include scenario specificity (such as a high-age debt collection scenario, i.e., a debt collection scenario where debts are owed for a long time), process specificity (such as complex operation process), etc. Thus, when the abnormal analysis dimension includes the number of conversation rounds being too long, it can be determined that the corresponding influencing factors include scenario specificity and process specificity.

[0075] In another possible implementation, the abnormal analysis dimension may also be input into a pre-trained fifth model to obtain the influencing factor output by the fifth model.

[0076] Among them, the fifth model is used to calculate the correlation between the abnormal analysis dimension and multiple preset influencing factors, and output the preset influencing factors whose correlation is greater than the preset correlation threshold as the influencing factors corresponding to the abnormal analysis dimension.

[0077] (2) Determine data analysis requirements based on the anomaly analysis dimensions and influencing factors.

[0078] Specifically, the intentions of the abnormal analysis dimension and the influencing factor can be identified respectively to obtain corresponding data analysis requirements.

[0079] For example, if the influencing factors include scenario specificity, then after intent recognition of the influencing factors, the corresponding data analysis requirements can be obtained, which are the dialogue scenarios for analyzing the target data; if the influencing factors include process specificity, then after intent recognition of the influencing factors, the corresponding data analysis requirements can be obtained, which are the operation processes contained in the analysis target data.

[0080] In other embodiments, in order to further improve the comprehensiveness and reliability of data analysis, the normal text data in the plurality of first text data may be further analyzed to obtain data distribution information corresponding to the normal text data. The data distribution information is used to characterize the data distribution span of the normal text data in the second preset analysis dimension. The data analysis requirements are determined in combination with the data distribution information. Specifically, the following steps may be included: (1) Obtaining second description information of normal text data in the plurality of first text data in a second preset analysis dimension.

[0081] Among them, the normal text data is the text data other than the abnormal text data in the multiple first text data. The second preset analysis dimension can be the same as or different from the first preset analysis dimension. Since the abnormal text data is usually the focus of data analysis, a more conventional analysis dimension can be selected from the preset analysis dimensions as the second preset analysis dimension to reduce the amount of data processing. Accordingly, the second description information can be the description information corresponding to the second preset analysis dimension.

[0082] (2) Determine data distribution information of the normal text data in a second preset analysis dimension based on the second description information of the normal text data.

[0083] For example, when the second preset analysis dimension includes a sentence length value, the normal text data with the highest and lowest conversation rounds can be determined based on the second description information, and the data distribution span of the normal text data on the second preset analysis dimension, that is, the data distribution information, can be determined based on the maximum and minimum values ​​of the conversation rounds. For example, if the maximum value of the conversation rounds is 10 and the minimum value of the conversation rounds is 4, then the data distribution span of the normal text data on the current second preset analysis dimension is the number of conversation rounds between 4 and 10 rounds. By statistics on the data distribution information of normal text data, it is possible to understand the data distribution spans allowed by normal text data on different analysis dimensions, which is convenient for subsequent data analysis of abnormal text data.

[0084] (3) Determine data analysis requirements based on the anomaly analysis dimension and data analysis information.

[0085] Specifically, the intention may be identified on the abnormal analysis dimension and the data distribution information to obtain the data analysis requirement.

[0086] For example, if the data distribution information indicates that the number of conversation rounds is between 4 and 10 rounds, after performing intent recognition on the data distribution information, the corresponding data analysis requirement can be obtained to determine whether the number of conversation rounds of the second text data is between 4 and 10 rounds.

[0087] In this way, by combining abnormal text data and normal text data to determine data analysis requirements, the comprehensiveness of subsequent data analysis can be improved.

[0088] In other embodiments, it is considered that most of the above data analysis requirements are texts with relatively simple semantics, and in actual application scenarios there may be semantically unrelated text data, that is, the meanings expressed by the contexts in the text data are not close. For example, if a customer asks question A, but the agent answers question B, then the semantics between the two are unrelated. Therefore, in order to further improve the comprehensiveness of data analysis, semantic analysis can also be added, that is, determining the semantic relevance between the contexts in the second text data as a data analysis requirement, so that in the subsequent data analysis process, the semantic logic of the second text data can be analyzed to determine whether there is a problem of unclear logic.

[0089] It should be noted that the above Figures 1 to 3 The various embodiments may be implemented separately or in combination. For example, the data analysis requirements may be determined based on the abnormal analysis dimensions, influencing factors, and data analysis information. This disclosure does not specifically limit this.

[0090] The following is a detailed description of the data analysis of the second text data according to the data analysis requirement in the above step S203. Specifically, the following steps may be included: (1) The data analysis requirement is input into a pre-trained first model to obtain a first analysis code output by the first model.

[0091] The first model can be a code generation model in the related art. By inputting the data analysis requirements into the first model, the first analysis code output by the first model for realizing the data analysis requirements can be obtained. The function of the first model can be understood as "translation", that is, translating the data analysis requirements into the corresponding code language.

[0092] In this step, the data analysis requirements can also be classified, and similar and dependent data analysis requirements can be classified into one category. Then, the corresponding first analysis code can be determined for each category of data analysis requirements. In this way, similar and dependent data analysis requirements can be analyzed simultaneously, making the entire analysis process smoother and faster.

[0093] (2) Determine the data analysis code based on the first analysis code.

[0094] In a possible implementation, the first analysis code may be directly used as the data analysis code.

[0095] In another possible implementation, in order to improve the operational reliability of the data analysis code, the first analysis code may be further screened, and the screened first analysis code may be used as the data analysis code.

[0096] Specifically, the sample data and the sample analysis results corresponding to the sample data can be obtained. Then, by running the first candidate analysis code, data analysis is performed on the sample data to obtain the data analysis results corresponding to the sample data. After that, the loss value (loss) between the sample analysis result and the data analysis result is determined. Among them, the similarity between the sentence vector of the sample analysis result and the sentence vector of the data analysis result can be calculated and used as the loss value. Then, the first analysis code whose loss value is less than or equal to the preset loss value is used as the second analysis code. And based on the second analysis code, the data analysis code is determined. Among them, the second analysis code can be used as the data analysis code. In this way, by running the first analysis code and comparing it with the sample analysis result, the second analysis code with the most accurate running result is selected from the first analysis code as the data analysis code, thereby improving the running reliability of the data analysis code.

[0097] Furthermore, in order to make the data analysis code have the characteristics of few lines of code script, strong reusability and high running efficiency, in some embodiments, the second analysis code can also be input into the pre-trained second model to obtain the code level corresponding to the second analysis code output by the second model. And the second analysis code whose code level meets the preset level requirement is used as the data analysis code.

[0098] The second model is used to analyze the input code and then evaluate the code level. The second model can be obtained by training a large number of pre-built code sample sets, and the code sample sets include code scripts and code levels corresponding to the code scripts.

[0099] In this way, by further screening from the second analysis code, a data analysis code with strong running reliability, few code script lines, strong reusability and high running efficiency can be obtained, thereby improving the efficiency of subsequent data analysis.

[0100] (3) Performing data analysis on the second text data according to the data analysis code.

[0101] For example, the data analysis code is compiled to obtain a corresponding data analysis program, and the data analysis is performed on the second text data through the data analysis program.

[0102] By adopting the above method, based on the first description information corresponding to the first preset analysis dimension of each first text data, data detection is performed on multiple first text data, and abnormal text data in the data and abnormal analysis dimensions corresponding to the abnormal text data are automatically mined. The abnormal analysis dimension is used as an analytical breakthrough for data analysis to obtain the corresponding data analysis requirements, enriching the idea of ​​data analysis and mining. Then, according to the data analysis requirements, data analysis is performed on the second text data to be analyzed. In this way, the downstream model can perform data analysis on the second text data to be analyzed based on the data analysis requirements before, during, or after training, thereby ensuring the reliability and credibility of the data analysis results. At the same time, it also provides support for the data analysis results for the subsequent training optimization of downstream models.

[0103] Figure 4 is a block diagram of a data processing device according to an exemplary embodiment. Figure 4 As shown, the device 300 includes: The acquisition module 301 is configured to determine, for each first text data among a plurality of first text data, first description information of the first text data in a first preset analysis dimension; The determination module 302 is configured to determine abnormal text data and an abnormal analysis dimension corresponding to the abnormal text data from the plurality of first text data according to the first description information of each first text data; the abnormal analysis dimension refers to at least one of the first preset analysis dimensions; The analysis module 303 is configured to determine a data analysis requirement according to the abnormal analysis dimension, and perform data analysis on the second text data according to the data analysis requirement.

[0104] Optionally, the determination module 302 is configured to determine the first similarity between each two first text data based on the first description information of each first text data; determine abnormal text data from the multiple first text data based on the first similarity; and determine the abnormal analysis dimension corresponding to the abnormal text data based on the first similarity.

[0105] Optionally, the number of the first preset analysis dimensions is multiple; the determination module 302 is configured to determine, for each first text data, the sum of similarities corresponding to the first text data based on the first similarities corresponding to the first text data under each first preset analysis dimension; and determine, among the multiple first text data, the first text data whose sum of similarities is less than or equal to the first preset similarity threshold as abnormal text data.

[0106] Optionally, the determination module 302 is configured to obtain a dimension weight corresponding to each first preset analysis dimension; based on the dimension weight, perform a preset operation on the first similarity corresponding to the first text data under each first preset analysis dimension to obtain the sum of similarities corresponding to the first text data.

[0107] Optionally, there are multiple first preset analysis dimensions; the determination module 302 is configured to determine, for each first preset analysis dimension, a first similarity related to the abnormal text data in the first similarity corresponding to the first preset analysis dimension as the second similarity; and determine the abnormal analysis dimension corresponding to the abnormal text data based on the second similarity of the abnormal text data under each first preset analysis dimension.

[0108] Optionally, the determination module 302 is configured to determine, for each first preset analysis dimension, a quantitative proportion of the third similarity in the second similarity of the first preset analysis dimension, the third similarity being a second similarity in the second similarity under the first preset analysis dimension that is less than or equal to a second preset similarity threshold; and determine the first preset analysis dimension whose quantitative proportion is greater than or equal to the preset quantitative proportion as an abnormal analysis dimension.

[0109] Optionally, the analysis module 303 is configured to determine the influencing factors of the abnormality of the abnormal text data according to the abnormality analysis dimension; and determine the data analysis requirements according to the abnormality analysis dimension and the influencing factors.

[0110] Optionally, the analysis module 303 is configured to obtain second description information of normal text data in the multiple first text data in a second preset analysis dimension; determine data distribution information of the normal text data in the second preset analysis dimension based on the second description information of the normal text data; and determine data analysis requirements based on the abnormal analysis dimension and the data distribution information.

[0111] Optionally, the analysis module 303 is configured to input the data analysis requirement into a pre-trained first model to obtain a first analysis code output by the first model; determine the data analysis code based on the first analysis code; and perform data analysis on the second text data based on the data analysis code.

[0112] Optionally, the analysis module 303 is configured to obtain sample data and sample analysis results corresponding to the sample data; perform data analysis on the sample data to obtain data analysis results by running the first analysis code; determine a loss value between the sample analysis result and the data analysis result; if the loss value is less than or equal to a preset loss value, use the first analysis code as a second analysis code; and determine the data analysis code based on the second analysis code.

[0113] Optionally, the analysis module 303 is configured to input the second analysis code into a pre-trained second model to obtain a code level corresponding to the second analysis code output by the second model; and use the second analysis code whose code level meets the preset level requirements as the data analysis code.

[0114] By using the above-mentioned device, data detection is performed on multiple first text data based on the first description information corresponding to the first preset analysis dimension of each first text data, and abnormal text data in the data and abnormal analysis dimensions corresponding to the abnormal text data are automatically mined. The abnormal analysis dimension is used as an analytical breakthrough for data analysis to obtain the corresponding data analysis requirements, enriching the idea of ​​data analysis and mining. Then, according to the data analysis requirements, data analysis is performed on the second text data to be analyzed. In this way, the downstream model can perform data analysis on the second text data to be analyzed based on the data analysis requirements before, during, or after training, thereby ensuring the reliability and credibility of the data analysis results. At the same time, it also provides support for the data analysis results for the subsequent training optimization of downstream models.

[0115] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0116] Figure 5 4 is a block diagram of an electronic device 400 according to an exemplary embodiment. For example, the electronic device 400 may be provided as a server. Figure 5 The electronic device 400 includes a processor 422, which may be one or more, and a memory 432 for storing a computer program executable by the processor 422. The computer program stored in the memory 432 may include one or more modules each corresponding to a set of instructions. In addition, the processor 422 may be configured to execute the computer program to perform the above-mentioned data processing method.

[0117] In addition, the electronic device 400 may further include a power supply component 426 and a communication component 450, wherein the power supply component 426 may be configured to perform power management of the electronic device 400, and the communication component 450 may be configured to implement communication of the electronic device 400, for example, wired or wireless communication. In addition, the electronic device 400 may further include an input / output (I / O) interface 458. The electronic device 400 may operate based on an operating system stored in the memory 432.

[0118] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by the processor, the steps of the above-mentioned data processing method are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 432 including program instructions, and the above-mentioned program instructions can be executed by the processor 422 of the electronic device 400 to complete the above-mentioned data processing method.

[0119] In another exemplary embodiment, a computer program product is also provided. The computer program product includes a computer program executable by a programmable device, and the computer program has a code portion for executing the above-mentioned data processing method when executed by the programmable device.

[0120] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments. Within the technical concept of the present disclosure, a variety of simple modifications can be made to the technical solution of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.

[0121] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0122] In addition, various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A method for data processing, characterized in that: The method comprises: For each first text data among the plurality of first text data, determining first description information of the first text data in a first preset analysis dimension; According to the first description information of each first text data, determine abnormal text data and abnormal analysis dimensions corresponding to the abnormal text data from the plurality of first text data; the abnormal analysis dimension refers to at least one of the first preset analysis dimensions; Data analysis requirements are determined according to the abnormal analysis dimension, and data analysis is performed on the second text data according to the data analysis requirements.

2. The method according to claim 1, characterized in that The determining, according to the first description information of each first text data, abnormal text data and abnormal analysis dimensions corresponding to the abnormal text data from the plurality of first text data includes: Determine, according to the first description information of each first text data, a first similarity between every two first text data; According to the first similarity, abnormal text data is determined from the plurality of first text data, and according to the first similarity, an abnormality analysis dimension corresponding to the abnormal text data is determined.

3. The method according to claim 2, characterized in that The number of the first preset analysis dimensions is multiple; and determining abnormal text data from the multiple first text data according to the first similarity includes: For each first text data, based on the first similarity corresponding to the first text data under each first preset analysis dimension, determine the sum of similarities corresponding to the first text data; Among the plurality of first text data, first text data whose sum of similarities is less than or equal to a first preset similarity threshold is determined as abnormal text data.

4. The method according to claim 2, characterized in that: The number of the first preset analysis dimensions is multiple; and determining the abnormal analysis dimension corresponding to the abnormal text data according to the first similarity includes: For each first preset analysis dimension, determining, among the first similarities corresponding to the first preset analysis dimension, a first similarity related to the abnormal text data as a second similarity; According to the second similarity of the abnormal text data under each first preset analysis dimension, the abnormal analysis dimension corresponding to the abnormal text data is determined.

5. The method according to claim 4, characterized in that The determining, according to the second similarity of the abnormal text data under each first preset analysis dimension, the abnormal analysis dimension corresponding to the abnormal text data includes: For each first preset analysis dimension, determining a quantitative proportion of a third similarity in the second similarities of the first preset analysis dimension, the third similarity being a second similarity in the second similarities under the first preset analysis dimension that is less than or equal to a second preset similarity threshold; The first preset analysis dimension whose quantity proportion is greater than or equal to the preset quantity proportion is determined as the abnormal analysis dimension.

6. The method according to claim 1, characterized in that The determining of data analysis requirements according to the abnormal analysis dimension includes: Determine, according to the abnormality analysis dimension, the influencing factors of the abnormality of the abnormal text data; Determine data analysis requirements based on the abnormality analysis dimensions and the influencing factors.

7. The method according to claim 1, characterized in that The performing data analysis on the second text data according to the data analysis requirement includes: Inputting the data analysis requirement into a pre-trained first model to obtain a first analysis code output by the first model; Determine a data analysis code according to the first analysis code; The second text data is subjected to data analysis according to the data analysis code.

8. The method according to claim 7, characterized in that Determining a data analysis code according to the first analysis code includes: Acquire sample data and sample analysis results corresponding to the sample data; By running the first analysis code, data analysis is performed on the sample data to obtain a data analysis result; Determining a loss value between the sample analysis result and the data analysis result; If the loss value is less than or equal to a preset loss value, the first analysis code is used as a second analysis code; The data analysis code is determined according to the second analysis code.

9. A data processing device, characterized in that: The device comprises: An acquisition module is configured to determine, for each first text data among a plurality of first text data, first description information of the first text data in a first preset analysis dimension; A determination module is configured to determine abnormal text data and abnormal analysis dimensions corresponding to the abnormal text data from the plurality of first text data according to the first description information of each first text data; the abnormal analysis dimension refers to at least one of the first preset analysis dimensions; The analysis module is configured to determine data analysis requirements according to the abnormal analysis dimension, and perform data analysis on the second text data according to the data analysis requirements.

10. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 8.

Images