Unstructured data extraction method and device, electronic equipment and storage medium

Abstract

The invention provides an unstructured data extraction method and device, electronic equipment and a storage medium, and relates to the technical field of data processing.The method comprises the steps that unstructured text data is obtained, and entity standard attributes corresponding to the unstructured text data are determined; inputting the unstructured text data into the large structured analysis model to obtain a structured analysis result output by the large structured analysis model and classification label accuracy corresponding to the structured analysis result; determining data supplement attribute information and data candidate attribute information according to the entity standard attribute, the structured analysis result and the classification label accuracy; and determining a data extraction result corresponding to the unstructured text data according to the data supplement attribute information and the data candidate attribute information. According to the technical scheme, the data extraction result is comprehensively determined based on the structured analysis result obtained by the structured analysis large model and the classification label accuracy, and the extraction accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, electronic device and storage medium for extracting unstructured data. Background Art

[0002] For unstructured text, it is impossible to perform structured description, and operations and queries can only be processed as a piece of text. Therefore, text content extraction for unstructured text becomes particularly important.

[0003] At present, in the prior art, the text content extraction method for unstructured text usually adopts recurrent neural network or long short-term memory method to directly extract information. The disadvantage of these two models is that during the training process, the training time is positively correlated with the length of the text content. The longer the text is, the longer the model training time is. In addition, the longer the text is, the lower the extraction accuracy will be. Summary of the invention

[0004] The present invention provides an unstructured data extraction method, device, electronic device and storage medium, which are used to solve the defects in the prior art that the use experience of extraction is reduced and the accuracy of extraction is reduced when a recurrent neural network or a long short-term memory method is used for extraction. A structured analysis large model is constructed based on the structured analysis for text classification and information extraction of unstructured text data, and the unstructured text data is analyzed without considering the length of the unstructured text data to obtain structured analysis results and classification label accuracy. The data extraction results corresponding to the unstructured text data are further comprehensively determined by the structured analysis results and the classification label accuracy corresponding to the structured analysis results, thereby improving the accuracy of extraction.

[0005] The present invention provides an unstructured data extraction method, comprising the following steps.

[0006] Obtain unstructured text data and determine standard attributes of entities corresponding to the unstructured text data.

[0007] The unstructured text data is input into the structured analysis big model to obtain the structured analysis results output by the structured analysis big model and the accuracy of the classification labels corresponding to the structured analysis results; wherein the structured analysis big model is a model for text classification and information extraction of unstructured text data of unlimited length.

[0008] The data supplementary attribute information and data candidate attribute information are determined according to the standard attributes of the entity, the structured analysis results and the accuracy of the classification labels; wherein the supplementary attribute information refers to the attribute information determined after analyzing the structured analysis results based on the standard attributes of the entity, and the data candidate attribute information refers to the attribute information determined after analyzing the structured analysis results based on the accuracy of the classification labels.

[0009] The data extraction result corresponding to the unstructured text data is determined according to the data supplementary attribute information and the data candidate attribute information.

[0010] According to an unstructured data extraction method provided by the present invention, unstructured text data is obtained, including: obtaining initialization unstructured text data; cleaning the initialization unstructured text data to obtain unstructured text data.

[0011] According to an unstructured data extraction method provided by the present invention, the training process of the structured analysis big model includes the following: obtaining unstructured sample data; determining an unstructured training set and an unstructured test set based on the unstructured sample data; inputting the unstructured training set into the basic analysis big model to obtain the analysis result of the basic analysis big model input; fine-tuning the basic analysis big model based on the analysis result to obtain an initial structured analysis big model; inputting the unstructured test set into the initial structured analysis big model to obtain the evaluation result of the initial structured analysis big model output; and obtaining the structured analysis big model when the evaluation result meets a preset threshold.

[0012] According to an unstructured data extraction method provided by the present invention, unstructured text data is input into a structured analysis big model to obtain a structured analysis result output by the structured analysis big model and the accuracy of the classification label corresponding to the structured analysis result, including: inputting the unstructured text data into a text classifier in the structured analysis big model to obtain a text classification result output by the text classifier; inputting the text classification result into an information extractor in the structured analysis big model to obtain a structured analysis result output by the information extractor; inputting the structured analysis result into a label classifier in the structured analysis big model to obtain the accuracy of the classification label output by the label classifier.

[0013] According to an unstructured data extraction method provided by the present invention, data supplementary attribute information and data candidate attribute information are determined according to entity standard attributes, structured analysis results and classification label accuracy, including: determining whether the classification label in the structured analysis result belongs to the entity standard attribute; when the classification label in the structured analysis result belongs to the entity standard attribute, determining the classification label in the structured analysis result belonging to the entity standard attribute as data supplementary attribute information; when the classification label in the structured analysis result does not belong to the entity standard attribute, determining the data candidate attribute information according to the classification label in the structured analysis result that does not belong to the entity standard attribute and the classification label accuracy.

[0014] According to an unstructured data extraction method provided by the present invention, data candidate attribute information is determined based on the classification labels and classification label accuracy in the structured analysis results that do not belong to the standard attributes of the entity, including: determining whether the classification label accuracy corresponding to the classification labels in the structured analysis results that do not belong to the standard attributes of the entity is greater than or equal to a preset accuracy; wherein the preset accuracy is a pre-set accuracy for judging the accuracy of the classification labels; when it is determined that the classification label accuracy corresponding to the classification labels in the structured analysis results that do not belong to the standard attributes of the entity is greater than or equal to the preset accuracy, determining that the classification labels in the structured analysis results that do not belong to the standard attributes of the entity are data candidate attribute information.

[0015] According to an unstructured data extraction method provided by the present invention, the method further includes: when it is determined that the classification label accuracy corresponding to the classification label in the structured analysis results that do not belong to the standard attributes of the entity is less than a preset accuracy, determining that the classification label in the structured analysis results that do not belong to the standard attributes of the entity is abnormal, and determining abnormal information; wherein the abnormal information is used to prompt that the data extraction of the unstructured text data is abnormal.

[0016] The present invention also provides an unstructured data extraction device, comprising the following modules.

[0017] The data acquisition module is used to acquire unstructured text data and determine the standard attributes of entities corresponding to the unstructured text data.

[0018] The result output module is used to input unstructured text data into the structured analysis big model to obtain the structured analysis results output by the structured analysis big model and the accuracy of the classification labels corresponding to the structured analysis results; wherein, the structured analysis big model is a model for text classification and information extraction of unstructured text data of unlimited length.

[0019] The information determination module is used to determine data supplementary attribute information and data candidate attribute information based on entity standard attributes, structured analysis results and classification label accuracy; wherein, supplementary attribute information refers to attribute information determined after analyzing the structured analysis results based on entity standard attributes, and data candidate attribute information refers to attribute information determined after analyzing the structured analysis results based on classification label accuracy.

[0020] The result determination module is used to determine the data extraction result corresponding to the unstructured text data according to the data supplementary attribute information and the data candidate attribute information.

[0021] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-mentioned unstructured data extraction methods is implemented.

[0022] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the computer program implements any of the above-mentioned unstructured data extraction methods.

[0023] The present invention also provides a computer program product, comprising a computer program, and when the computer program is executed by a processor, the computer program implements any of the above-mentioned unstructured data extraction methods.

[0024] The present invention provides an unstructured data extraction method, device, electronic device and storage medium, which obtain unstructured text data and determine the standard attributes of entities corresponding to the unstructured text data; input the unstructured text data into a structured analysis big model to obtain the structured analysis results output by the structured analysis big model and the classification label accuracy corresponding to the structured analysis results; wherein the structured analysis big model is a model for text classification and information extraction for unstructured text data of unlimited length; determine data supplementary attribute information and data candidate attribute information according to the standard attributes of entities, the structured analysis results and the classification label accuracy; wherein the supplementary attribute information refers to the attribute information determined after analyzing the structured analysis results based on the standard attributes of entities, and the data candidate attribute information refers to the attribute information after analyzing the structured analysis results based on the classification label accuracy; determine the data extraction results corresponding to the unstructured text data according to the data supplementary attribute information and the data candidate attribute information. The technical solution of the present invention is used to solve the defects in the prior art that the use of recurrent neural networks or long short-term memory methods for extraction leads to a decrease in the usability of extraction and a decrease in the accuracy of extraction, and realizes a structured analysis large model based on the construction for text classification and information extraction of unstructured text data, analyzes the unstructured text data, obtains structured analysis results and classification label accuracy, and further comprehensively determines the data extraction results corresponding to the unstructured text data through the structured analysis results and classification label accuracy, thereby improving the accuracy of extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 It is a flow chart of the unstructured data extraction method provided by the present invention.

[0027] Figure 2 It is a structural schematic diagram of the unstructured data extraction device provided by the present invention.

[0028] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] Combine the following Figure 1 The unstructured data extraction method provided by the present invention is described. The unstructured data extraction method provided by the present invention can be applicable to the extraction of unstructured data. The execution subject of the method can be an electronic device or an unstructured data extraction device arranged in the electronic device. The unstructured data extraction device can be implemented by software, hardware or a combination of both. Figure 1 is a flow chart of the unstructured data extraction method provided by the present invention, such as Figure 1 As shown, the method includes the following steps 101, 102, 103 and 104.

[0031] Step 101: Obtain unstructured text data and determine standard attributes of entities corresponding to the unstructured text data.

[0032] In this step, unstructured text data refers to customer entity model data used to carry customer data. The entity standard attributes corresponding to unstructured text data mainly include customer name, customer follow-up person, follow-up priority, customer level, industry, customer source, business category, etc.; the customer entity large text attributes corresponding to unstructured text data include business profile and operation information. Business profile and operation information are text domain attributes, which are used by users to fill in the customer's business introduction and operation data. There is no unified regulation on the content specifications, and different users will have different description methods of content input. Business profile and operation information carry a large unstructured text data. For example, the business profile can be focused on the research and development, production and sales of video surveillance products, providing comprehensive security solutions, including cameras, video management systems and intelligent analysis tools. XX company is committed to improving XX security level through innovative technologies and serving various markets, such as urban security, traffic monitoring, building security, etc. Operational information can be committed to helping enterprises optimize customer relationships and improve sales efficiency through innovative customer relationship management (CRM) solutions. XX Company provides comprehensive customer management tools, including sales management, customer service and marketing functions, and uses advanced data analysis technology to provide in-depth sales and customer data analysis to help companies make data-based decisions. In addition, XX Company also provides mobile applications to ensure that sales personnel can access and manage customer information anytime and anywhere, thereby improving work efficiency, which is not limited in this embodiment.

[0033] Specifically, unstructured text data of the target customer entity is obtained, which may be, for example, a business profile and operational information, and standard entity attributes corresponding to the unstructured text data are determined, which may be, for example, a customer name, a customer follow-up person, a follow-up priority, a customer level, an industry, a customer source, and a business category.

[0034] For example, customer name, customer follow-up person, follow-up priority, customer level, industry, customer source and business category all have attribute definitions of data type, data length and data uniqueness, which are not limited in this embodiment. The entity standard attributes corresponding to the unstructured text data are shown in Table 1, which shows the specific content of the entity standard attributes corresponding to the unstructured text data, which are not limited in this embodiment.

[0035] Table 1

[0036] In a specific implementation, obtaining unstructured text data includes: obtaining initialization unstructured text data; and cleaning the initialization unstructured text data to obtain unstructured text data.

[0037] Specifically, after the initialization unstructured text data is obtained, the initialization unstructured text data is further cleaned to ensure the quality and quantity of the unstructured text data after clear processing.

[0038] The advantage of this setting is that it is easy to apply to the subsequent structured analysis model and improve the accuracy of the structured analysis model in extracting unstructured text data.

[0039] Step 102: input the unstructured text data into the structured analysis model to obtain the structured analysis results output by the structured analysis model and the accuracy of the classification labels corresponding to the structured analysis results.

[0040] In this step, the structured analysis large model is a model used to perform text classification and information extraction on unstructured text data of unlimited length.

[0041] Specifically, after obtaining the unstructured text data, the unstructured text data is input into the structured analysis model to obtain the structured analysis results output by the structured analysis model and the accuracy of the classification labels corresponding to the structured analysis results.

[0042] In a specific embodiment, the training process of the structured analysis big model includes the following: obtaining unstructured sample data; determining an unstructured training set and an unstructured test set based on the unstructured sample data; inputting the unstructured training set into the basic analysis big model to obtain the analysis results of the basic analysis big model input; fine-tuning the basic analysis big model based on the analysis results to obtain an initial structured analysis big model; inputting the unstructured test set into the initial structured analysis big model to obtain the evaluation results of the initial structured analysis big model output; when the evaluation results meet a preset threshold, a structured analysis big model is obtained.

[0043] In this step, the basic analysis model can be, for example, a large model of the deep learning framework PyTorch (an open source deep learning framework for machine learning and deep learning) loaded with the Bidirectional Encoder Representations from Transformers (BERT). BERT is a pre-trained language model for natural language processing, and the model parameters are obtained by pre-training on a large amount of text data.

[0044] The preset threshold is a pre-set threshold used to determine whether the initial structured analysis large model has been trained, and this embodiment does not limit this.

[0045] Specifically, the initialized unstructured sample data is obtained, and the data is annotated on the initialized unstructured sample data to obtain the unstructured sample data, and then the unstructured sample data is divided into an unstructured training set and an unstructured test set, and the basic analysis big model is trained by the unstructured training set to obtain the initial structured analysis big model, and then the unstructured test set is input into the initial structured analysis big model, and the evaluation result of the initial structured analysis big model output is obtained through customized loop training, and then the evaluation result is used to evaluate whether the initial structured analysis big model is trained, and when the training is not completed, the initial structured analysis big model is continuously trained in a loop based on the evaluation result until the evaluation result meets the preset threshold value, and then the structured analysis big model is obtained.

[0046] Exemplarily, the initialized unstructured sample data is annotated to obtain the unstructured sample data. For example, when the initialized unstructured sample data is as follows, Company A is committed to helping companies optimize customer relationships and improve sales efficiency through innovative CRM solutions. As the leading CRM provider in XX, Company A provides comprehensive customer management tools, including sales management, customer service, and marketing functions. Using advanced data analysis technology, it provides in-depth sales and customer data analysis to help companies make data-based decisions. In addition, Company A also provides mobile applications to ensure that sales staff can access and manage customer information anytime, anywhere, thereby improving work efficiency. Then, the initialized unstructured sample data is labeled, and the obtained unstructured sample data can be, for example, {"Business Category": "CRM Solution"}, "Key Points": {"Main Business": "Helping companies optimize customer relationships and improve sales efficiency", "Market Position": "XX's leading CRM provider", "Product Type": ["Sales Management", "Customer Service", "Marketing Function"]}, "Description": "Company A is committed to helping companies optimize customer relationships and improve sales efficiency through innovative CRM solutions. As XX's leading CRM provider, the company provides comprehensive customer management tools and uses advanced data analysis technology to support corporate decision-making, ultimately improving work efficiency."}.

[0047] The advantage of this setting is that through cyclic training, a structured analysis model can be obtained, which can improve the accuracy of the structured analysis model in analyzing and extracting unstructured text data.

[0048] In a specific embodiment, unstructured text data is input into a structured analysis big model to obtain structured analysis results output by the structured analysis big model and the accuracy of classification labels corresponding to the structured analysis results, including: inputting the unstructured text data into a text classifier in the structured analysis big model to obtain a text classification result output by the text classifier; inputting the text classification result into an information extractor in the structured analysis big model to obtain a structured analysis result output by the information extractor; inputting the structured analysis result into a label classifier in the structured analysis big model to obtain the accuracy of classification labels output by the label classifier.

[0049] In this step, the text classifier is used to perform text classification on unstructured text data, the information extractor is used to perform structured analysis on the text classification results, and the label classifier is used to calculate the classification label accuracy of the structured analysis results.

[0050] Specifically, the unstructured text data is input into the text classifier in the structured analysis model to obtain the text classification result output by the text classifier; the text classification result is input into the information extractor in the structured analysis model to obtain the structured analysis result output by the information extractor; the structured analysis result is input into the label classifier in the structured analysis model to obtain the classification label accuracy output by the label classifier.

[0051] For example, unstructured text data can be that Company B focuses on the research and development, production and sales of video surveillance products, and provides comprehensive security solutions, including cameras, video management systems and intelligent analysis tools. Company B is committed to improving the security level of XX through innovative technologies and serving various markets, such as urban security, traffic monitoring, building security, etc. It has a wide customer base and sales network within the scope of XX, and is committed to providing users with efficient and safe monitoring and management solutions. Input into the structured analysis big model, the structured analysis results output by the structured analysis big model can be, for example, {"Business Category": "Video Surveillance Products and Security Solutions"}, "Key Points": {"Main Business": "Research and Development, Production and Sales of Video Surveillance Products", "Product Type": ["Camera", "Video Management System", "Intelligent Analysis Tools"], "Service Market": ["Urban Security", "Traffic Monitoring", "Building Security"], "Customer Base": "A Wide Customer Base and Sales Network within the XX Scope"}, "Description": "Company B focuses on the research and development, production and sales of video surveillance products, and is committed to improving the security level of XX through innovative technologies and providing efficient and safe monitoring and management solutions."}. Input the structured analysis results into the label classifier in the structured analysis large model to obtain the classification label accuracy output by the label classifier. The classification label accuracy can be, for example, {"Label accuracy": {"Business category": {"Accuracy": 90},"Key point": {"Main business": {"Accuracy": 90},"Product type": {"Accuracy": 95},"Service market": {"Accuracy": 100},"Customer base": {"Accuracy": 80}},"Description": {"Accuracy": 85}}}.

[0052] Step 103: Determine data supplementary attribute information and data candidate attribute information according to entity standard attributes, structured analysis results, and classification label accuracy.

[0053] In this step, the supplementary attribute information refers to the attribute information determined after analyzing the structured analysis results based on the standard attributes of the entity, and the data candidate attribute information refers to the attribute information determined after analyzing the structured analysis results based on the accuracy of the classification labels.

[0054] Specifically, after obtaining the structured analysis results and the classification label accuracy, the data supplementary attribute information and the data candidate attribute information are determined according to the entity standard attributes, the structured analysis results and the classification label accuracy.

[0055] In a specific embodiment, data supplementary attribute information and data candidate attribute information are determined based on entity standard attributes, structured analysis results and classification label accuracy, including: determining whether the classification labels in the structured analysis results belong to the entity standard attributes; in the case where the classification labels in the structured analysis results belong to the entity standard attributes, determining the classification labels in the structured analysis results that belong to the entity standard attributes as data supplementary attribute information; in the case where the classification labels in the structured analysis results do not belong to the entity standard attributes, determining the data candidate attribute information based on the classification labels in the structured analysis results that do not belong to the entity standard attributes and the classification label accuracy.

[0056] In this step, the classification labels in the structured analysis results may be, for example, business categories, key points, descriptions, etc., which are not limited in this embodiment.

[0057] Specifically, determine whether the classification labels in the structured analysis results belong to the standard attributes of the entity. If the classification labels in the structured analysis results belong to the standard attributes of the entity, determine the classification labels in the structured analysis results that belong to the standard attributes of the entity as data supplementary attribute information; if the classification labels in the structured analysis results do not belong to the standard attributes of the entity, determine the data candidate attribute information based on the classification labels in the structured analysis results that do not belong to the standard attributes of the entity and the accuracy of the classification labels.

[0058] Exemplarily, the standard attributes of an entity include business categories and operating information, and it is determined that the classification tags in the structured analysis results include business categories, operating information, and service markets. Then, the business categories and operating information are determined as data supplementary attribute information. Furthermore, it is determined that the classification tags in the structured analysis results, such as the service market, do not belong to the standard attributes of the entity. The accuracy of the classification tags corresponding to the service market is determined, and the data candidate attribute information is determined based on the classification tags in the structured analysis results that do not belong to the standard attributes of the entity and the accuracy of the classification tags.

[0059] In a specific embodiment, data candidate attribute information is determined based on classification labels and classification label accuracy in structured analysis results that do not belong to standard attributes of an entity, including: determining whether the classification label accuracy corresponding to the classification labels in the structured analysis results that do not belong to standard attributes of the entity is greater than or equal to a preset accuracy; wherein the preset accuracy is a pre-set accuracy for judging the accuracy of classification labels; when it is determined that the classification label accuracy corresponding to the classification labels in the structured analysis results that do not belong to standard attributes of the entity is greater than or equal to the preset accuracy, determining that the classification labels in the structured analysis results that do not belong to standard attributes of the entity are data candidate attribute information.

[0060] In this step, the preset accuracy is a preset accuracy for determining the accuracy of the classification label, for example, it may be 90%, which is not limited in this embodiment.

[0061] Specifically, determine whether the accuracy of the classification label corresponding to the classification label in the structured analysis results that do not belong to the standard attributes of the entity is greater than or equal to the preset accuracy; wherein the preset accuracy is a pre-set accuracy for judging the accuracy of the classification label; when it is determined that the accuracy of the classification label corresponding to the classification label in the structured analysis results that do not belong to the standard attributes of the entity is greater than or equal to the preset accuracy, determine that the classification label in the structured analysis results that do not belong to the standard attributes of the entity is data candidate attribute information.

[0062] Exemplarily, determine that the service market does not belong to the classification labels in the structured analysis results of the entity's standard attributes, determine whether the accuracy of the classification labels corresponding to the service market is greater than or equal to the preset accuracy, and when it is determined that the accuracy of the classification labels corresponding to the classification labels in the structured analysis results that do not belong to the entity's standard attributes is greater than or equal to the preset accuracy, determine that the classification labels in the structured analysis results that do not belong to the entity's standard attributes are data candidate attribute information.

[0063] In a specific embodiment, it also includes: when it is determined that the classification label accuracy corresponding to the classification label in the structured analysis results that do not belong to the standard attributes of the entity is less than a preset accuracy, determining that the classification label in the structured analysis results that do not belong to the standard attributes of the entity is abnormal, and determining abnormal information; wherein the abnormal information is used to prompt data extraction abnormality of the unstructured text data.

[0064] In this step, the exception information is used to prompt the unstructured text data for data extraction exceptions.

[0065] Specifically, when it is determined that the classification label accuracy corresponding to the classification label in the structured analysis results that do not belong to the standard attributes of the entity is less than the preset accuracy, it is determined that the classification label in the structured analysis results that do not belong to the standard attributes of the entity is abnormal, and the abnormal information is determined; through the abnormal information, the analyst is prompted that the data extraction of the unstructured text data is abnormal.

[0066] When extracting unstructured text data, timely grasp the data extraction status and improve the analyst's user experience.

[0067] Step 104: Determine the data extraction result corresponding to the unstructured text data according to the data supplementary attribute information and the data candidate attribute information.

[0068] Specifically, after the data supplement attribute information and the data candidate attribute information are determined, a data extraction result corresponding to the unstructured text data is jointly generated through a collection of the data supplement attribute information and the data candidate attribute information.

[0069] The advantage of this setting is that the unstructured text data is analyzed using a large structured analysis model without considering the length of the text, and after obtaining the structured analysis results and the accuracy of the classification labels corresponding to the structured analysis results, the data supplementary attribute information and the data candidate attribute information are further determined based on the structured analysis results and the accuracy of the classification labels corresponding to the structured analysis results. Finally, the data extraction result is determined by comprehensively determining the data supplementary attribute information and the data candidate attribute information, thereby improving the accuracy of extracting unstructured text data and solving the problem of directly extracting information using recurrent neural networks or long short-term memory methods in the prior art. The disadvantage of these two models is that during the training process, the length of the training time is positively correlated with the length of the text content. The longer the text length, the longer the model training time, and the longer the text, the lower the extraction accuracy.

[0070] The present invention provides an unstructured data extraction method, which obtains unstructured text data and determines the entity standard attributes corresponding to the unstructured text data; inputs the unstructured text data into a structured analysis big model to obtain the structured analysis results output by the structured analysis big model and the classification label accuracy corresponding to the structured analysis results; wherein the structured analysis big model is a model for text classification and information extraction for unstructured text data of unlimited length; determines data supplementary attribute information and data candidate attribute information according to the entity standard attributes, the structured analysis results and the classification label accuracy; wherein the supplementary attribute information refers to the attribute information determined after analyzing the structured analysis results based on the entity standard attributes, and the data candidate attribute information refers to the attribute information after analyzing the structured analysis results based on the classification label accuracy; determines the data extraction result corresponding to the unstructured text data according to the data supplementary attribute information and the data candidate attribute information. The technical solution of the present invention is used to solve the defects in the prior art that the use of recurrent neural networks or long short-term memory methods for extraction leads to a decrease in the usability of extraction and a decrease in the accuracy of extraction, and realizes a structured analysis large model based on the construction for text classification and information extraction of unstructured text data, analyzes the unstructured text data, obtains structured analysis results and classification label accuracy, and further comprehensively determines the data extraction results corresponding to the unstructured text data through the structured analysis results and classification label accuracy, thereby improving the accuracy of extraction.

[0071] The unstructured data extraction device provided by the present invention is described below. The unstructured data extraction device described below and the unstructured data extraction method described above can be referred to each other.

[0072] Figure 2 is a schematic diagram of the structure of the unstructured data extraction device provided by the present invention, referring to Figure 2 As shown, the unstructured data extraction device 200 includes: a data acquisition module 201, a result output module 202, an information determination module 203 and a result determination module 204.

[0073] The data acquisition module 201 is used to acquire unstructured text data and determine the standard attributes of entities corresponding to the unstructured text data.

[0074] The result output module 202 is used to input the unstructured text data into the structured analysis big model to obtain the structured analysis results output by the structured analysis big model and the accuracy of the classification labels corresponding to the structured analysis results; wherein the structured analysis big model is a model for text classification and information extraction of unstructured text data of unlimited length.

[0075] The information determination module 203 is used to determine data supplementary attribute information and data candidate attribute information based on the standard attributes of the entity, the structured analysis results and the accuracy of the classification label; wherein the supplementary attribute information refers to the attribute information determined after analyzing the structured analysis results based on the standard attributes of the entity, and the data candidate attribute information refers to the attribute information determined after analyzing the structured analysis results based on the accuracy of the classification label.

[0076] The result determination module 204 is used to determine the data extraction result corresponding to the unstructured text data according to the data supplementary attribute information and the data candidate attribute information.

[0077] In an exemplary embodiment, the data acquisition module 201 is specifically used to: acquire initialization unstructured text data; and perform cleaning processing on the initialization unstructured text data to obtain unstructured text data.

[0078] In an exemplary embodiment, the device further includes: a model training module. The model training module is used to: obtain unstructured sample data; determine an unstructured training set and an unstructured test set according to the unstructured sample data; input the unstructured training set into the basic analysis big model to obtain the analysis result of the basic analysis big model input; fine-tune the basic analysis big model based on the analysis result to obtain an initial structured analysis big model; input the unstructured test set into the initial structured analysis big model to obtain the evaluation result output by the initial structured analysis big model; and obtain the structured analysis big model when the evaluation result meets the preset threshold.

[0079] In an exemplary embodiment, the result output module 202 is specifically used to: input unstructured text data into a text classifier in a structured analysis large model to obtain a text classification result output by the text classifier; input the text classification result into an information extractor in the structured analysis large model to obtain a structured analysis result output by the information extractor; input the structured analysis result into a label classifier in the structured analysis large model to obtain the classification label accuracy output by the label classifier.

[0080] In an exemplary embodiment, the information determination module 203 is specifically used to: determine whether the classification labels in the structured analysis results belong to the standard attributes of the entity; if the classification labels in the structured analysis results belong to the standard attributes of the entity, determine the classification labels in the structured analysis results that belong to the standard attributes of the entity as data supplementary attribute information; if the classification labels in the structured analysis results do not belong to the standard attributes of the entity, determine the data candidate attribute information based on the classification labels in the structured analysis results that do not belong to the standard attributes of the entity and the accuracy of the classification labels.

[0081] In an example embodiment, the information determination module 203 determines data candidate attribute information based on the classification labels and classification label accuracy in the structured analysis results that do not belong to the standard attributes of the entity, and is specifically used to: determine whether the classification label accuracy corresponding to the classification labels in the structured analysis results that do not belong to the standard attributes of the entity is greater than or equal to a preset accuracy; wherein the preset accuracy is a pre-set accuracy for judging the accuracy of the classification labels; when it is determined that the classification label accuracy corresponding to the classification labels in the structured analysis results that do not belong to the standard attributes of the entity is greater than or equal to the preset accuracy, determine that the classification labels in the structured analysis results that do not belong to the standard attributes of the entity are data candidate attribute information.

[0082] In an example embodiment, the information determination module 203 determines data candidate attribute information based on the classification labels and classification label accuracy in the structured analysis results that do not belong to the standard attributes of the entity, and is also used to: when it is determined that the classification label accuracy corresponding to the classification label in the structured analysis results that do not belong to the standard attributes of the entity is less than a preset accuracy, determine that the classification label in the structured analysis results that do not belong to the standard attributes of the entity is abnormal, and determine abnormal information; wherein the abnormal information is used to prompt data extraction abnormalities of unstructured text data.

[0083] The device of this embodiment can be used to execute the method of any embodiment in the unstructured data extraction method side embodiment. Its specific implementation process and technical effects are similar to those in the unstructured data extraction method side embodiment. For details, please refer to the detailed introduction in the unstructured data extraction method side embodiment, which will not be repeated here.

[0084] Figure 3 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 3 As shown, the electronic device may include: a processor 310 , a communications interface 320 , a memory 330 and a communication bus 340 , wherein the processor 310 , the communications interface 320 , and the memory 330 communicate with each other via the communication bus 340 . The processor 310 can call the logic instructions in the memory 330 to execute the unstructured data extraction method, which includes: obtaining unstructured text data and determining the standard attributes of the entity corresponding to the unstructured text data; inputting the unstructured text data into the structured analysis big model to obtain the structured analysis results output by the structured analysis big model and the classification label accuracy corresponding to the structured analysis results; wherein the structured analysis big model is a model for text classification and information extraction for unstructured text data of unlimited length; determining data supplementary attribute information and data candidate attribute information based on the standard attributes of the entity, the structured analysis results and the classification label accuracy; wherein the supplementary attribute information refers to the attribute information determined after analyzing the structured analysis results based on the standard attributes of the entity, and the data candidate attribute information refers to the attribute information after analyzing the structured analysis results based on the classification label accuracy; determining the data extraction result corresponding to the unstructured text data based on the data supplementary attribute information and the data candidate attribute information.

[0085] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0086] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the unstructured data extraction method provided by the above methods, which includes: obtaining unstructured text data and determining the standard attributes of the entity corresponding to the unstructured text data; inputting the unstructured text data into a structured analysis big model to obtain the structured analysis results output by the structured analysis big model and the classification label accuracy corresponding to the structured analysis results; wherein the structured analysis big model is a model for text classification and information extraction for unstructured text data of unlimited length; determining data supplementary attribute information and data candidate attribute information based on the standard attributes of the entity, the structured analysis results and the classification label accuracy; wherein the supplementary attribute information refers to the attribute information determined after analyzing the structured analysis results based on the standard attributes of the entity, and the data candidate attribute information refers to the attribute information after analyzing the structured analysis results based on the classification label accuracy; determining the data extraction result corresponding to the unstructured text data based on the data supplementary attribute information and the data candidate attribute information.

[0087] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the unstructured data extraction method provided by the above-mentioned methods, the method comprising: obtaining unstructured text data, and determining the standard attributes of the entity corresponding to the unstructured text data; inputting the unstructured text data into a structured analysis big model, and obtaining the structured analysis results output by the structured analysis big model and the classification label accuracy corresponding to the structured analysis results; wherein the structured analysis big model is a model for text classification and information extraction for unstructured text data of unlimited length; determining data supplementary attribute information and data candidate attribute information based on the standard attributes of the entity, the structured analysis results and the classification label accuracy; wherein the supplementary attribute information refers to the attribute information determined after analyzing the structured analysis results based on the standard attributes of the entity, and the data candidate attribute information refers to the attribute information after analyzing the structured analysis results based on the classification label accuracy; determining the data extraction result corresponding to the unstructured text data based on the data supplementary attribute information and the data candidate attribute information.

[0088] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0089] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for extracting unstructured data, characterized in that: include: Acquire unstructured text data, and determine entity standard attributes corresponding to the unstructured text data; Inputting the unstructured text data into a structured analysis model to obtain a structured analysis result output by the structured analysis model and the accuracy of classification labels corresponding to the structured analysis result; wherein the structured analysis model is a model for performing text classification and information extraction on the unstructured text data of unlimited length; Determine data supplementary attribute information and data candidate attribute information according to the entity standard attribute, the structured analysis result and the classification label accuracy; wherein the supplementary attribute information refers to attribute information determined after analyzing the structured analysis result based on the entity standard attribute, and the data candidate attribute information refers to attribute information determined after analyzing the structured analysis result based on the classification label accuracy; A data extraction result corresponding to the unstructured text data is determined according to the data supplementary attribute information and the data candidate attribute information.

2. The unstructured data extraction method according to claim 1, characterized in that: The obtaining of unstructured text data includes: Get initialized unstructured text data; The initialized unstructured text data is cleaned to obtain the unstructured text data.

3. The unstructured data extraction method according to claim 1, characterized in that: The training process of the structured analysis large model includes the following: Obtain unstructured sample data; Determining an unstructured training set and an unstructured test set according to the unstructured sample data; Inputting the unstructured training set into a basic analysis large model to obtain an analysis result of the basic analysis large model input; Fine-tune the basic analysis model based on the analysis results to obtain an initial structured analysis model; Inputting the unstructured test set into the initial structured analysis large model to obtain an evaluation result output by the initial structured analysis large model; When the evaluation result meets a preset threshold, the structured analysis large model is obtained.

4. The unstructured data extraction method according to claim 1, characterized in that: The step of inputting the unstructured text data into a structured analysis model to obtain a structured analysis result output by the structured analysis model and the accuracy of classification labels corresponding to the structured analysis result includes: Inputting the unstructured text data into the text classifier in the structured analysis large model to obtain a text classification result output by the text classifier; Inputting the text classification result into the information extractor in the structured analysis large model to obtain the structured analysis result output by the information extractor; The structured analysis result is input into the label classifier in the structured analysis large model to obtain the classification label accuracy output by the label classifier.

5. The unstructured data extraction method according to claim 1, characterized in that: The determining of data supplementary attribute information and data candidate attribute information according to the entity standard attribute, the structured analysis result and the classification label accuracy includes: Determining whether the classification label in the structured analysis result belongs to the entity standard attribute; In the case where the classification label in the structured analysis result belongs to the entity standard attribute, determining the classification label in the structured analysis result belonging to the entity standard attribute as the data supplementary attribute information; In the case that the classification label in the structured analysis result does not belong to the entity standard attribute, the data candidate attribute information is determined according to the classification label in the structured analysis result that does not belong to the entity standard attribute and the classification label accuracy.

6. The unstructured data extraction method according to claim 5, characterized in that: The determining the data candidate attribute information according to the classification label in the structured analysis result that does not belong to the entity standard attribute and the accuracy of the classification label includes: Determine whether the classification label accuracy corresponding to the classification label in the structured analysis result that does not belong to the standard attribute of the entity is greater than or equal to a preset accuracy; wherein the preset accuracy is a preset accuracy for determining the accuracy of the classification label; When it is determined that the classification label accuracy corresponding to the classification label in the structured analysis result that does not belong to the standard attribute of the entity is greater than or equal to the preset accuracy, the classification label in the structured analysis result that does not belong to the standard attribute of the entity is determined to be the data candidate attribute information.

7. The unstructured data extraction method according to claim 6, characterized in that: Also includes: When it is determined that the classification label accuracy corresponding to the classification label in the structured analysis result that does not belong to the standard attribute of the entity is less than the preset accuracy, it is determined that the classification label in the structured analysis result that does not belong to the standard attribute of the entity is abnormal, and the abnormal information is determined; wherein, the abnormal information is used to prompt that the data extraction of the unstructured text data is abnormal.

8. An unstructured data extraction device, characterized in that: include: A data acquisition module, used to acquire unstructured text data and determine the entity standard attributes corresponding to the unstructured text data; A result output module, used for inputting the unstructured text data into the structured analysis big model, obtaining the structured analysis results output by the structured analysis big model and the classification label accuracy corresponding to the structured analysis results; wherein the structured analysis big model is a model for text classification and information extraction of the unstructured text data of unlimited length; An information determination module, used to determine data supplementary attribute information and data candidate attribute information according to the entity standard attribute, the structured analysis result and the classification label accuracy; wherein the supplementary attribute information refers to attribute information determined after analyzing the structured analysis result based on the entity standard attribute, and the data candidate attribute information refers to attribute information determined after analyzing the structured analysis result based on the classification label accuracy; A result determination module is used to determine the data extraction result corresponding to the unstructured text data according to the data supplementary attribute information and the data candidate attribute information.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the unstructured data extraction method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the unstructured data extraction method according to any one of claims 1 to 7 is implemented.

Images