Unstructured data labeling method and device, electronic equipment and storage medium

Abstract

The invention provides an unstructured data labeling method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining unstructured data, recognizing the unstructured data, and obtaining at least one title; performing semantic annotation on all the titles to obtain first attribute information corresponding to the unstructured data; obtaining the number of titles, and determining a first target recognition range and a second target recognition range; performing semantic annotation on the corresponding unstructured data in the first target recognition range to obtain second attribute information corresponding to the unstructured data; performing statement splitting on the corresponding unstructured data in the second target recognition range to obtain at least one statement; inputting the at least one statement into a statement attribute discrimination model obtained through pre-training, and obtaining third attribute information corresponding to the unstructured data through processing of the statement attribute discrimination model; and correspondingly storing the unstructured data, the first attribute information, the second attribute information and the third attribute information into a database.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a method, device, electronic device and storage medium for labeling unstructured data. Background Art

[0002] With the rapid development of the economy and society, scientific research has become increasingly extensive and in-depth, accumulating a large amount of scientific research project data. These data include not only various research results, experimental data and technical literature, but also multi-dimensional information such as project management, funding and expert allocation.

[0003] However, a large amount of data is unstructured, and traditional data processing methods find it difficult to effectively extract and utilize this information. Summary of the invention

[0004] In view of this, the purpose of the present disclosure is to propose a method, device, electronic device and storage medium for labeling unstructured data to solve or partially solve the above-mentioned problems.

[0005] Based on the above purpose, the first aspect of the present disclosure provides a method for labeling unstructured data, the method comprising:

[0006] Acquire unstructured data, identify the unstructured data, and obtain at least one title;

[0007] Performing semantic annotation on all titles to obtain first attribute information corresponding to the unstructured data;

[0008] Acquire the number of titles, and determine the first target recognition range and the second target recognition range according to the number of titles;

[0009] Performing semantic annotation on the corresponding unstructured data within the first target recognition range to obtain second attribute information corresponding to the unstructured data;

[0010] Perform sentence splitting on the corresponding unstructured data within the second target recognition range to obtain at least one sentence;

[0011] Inputting the at least one sentence into a pre-trained sentence attribute discrimination model, and processing the sentence attribute discrimination model to obtain third attribute information corresponding to the unstructured data;

[0012] The unstructured data, the first attribute information, the second attribute information and the third attribute information are correspondingly stored in a database.

[0013] Specifically, the unstructured data is identified to obtain at least one title, including:

[0014] Traversing the unstructured data, and taking characters in the unstructured data that meet a preset format as titles;

[0015] Obtaining all paragraphs contained in the unstructured data, and determining the length of characters contained in each paragraph;

[0016] In response to the existence of a target paragraph whose character length is less than or equal to a preset length threshold, the target paragraph is used as a title.

[0017] Specifically, the semantic annotation of all titles to obtain the first attribute information corresponding to the unstructured data includes:

[0018] Obtain a preset synonym library, and compare each title in all titles with the preset synonym library;

[0019] For each title: in response to the title and the preset synonym library having an intersection, determining the content corresponding to the intersection as the initial first attribute information; obtaining a first target recognition attribute, determining a first word vector corresponding to the title and a second word vector corresponding to the first target recognition attribute, and calculating the similarity between the first word vector and the second word vector; in response to the similarity being greater than a similarity threshold, determining the content corresponding to the title as the initial first attribute information;

[0020] The initial first attribute information corresponding to all titles is counted to obtain the first attribute information corresponding to the unstructured data.

[0021] Specifically, the acquiring the number of titles and determining the first target recognition range and the second target recognition range according to the number of titles include:

[0022] Get the number of titles;

[0023] In response to the number of titles being greater than a preset number, determining that a first target identification range is from the beginning of the document of the unstructured data to a target title, and a second target identification range is from the target title to the end of the document of the unstructured data, wherein the target titles are titles that are in sequence starting from the beginning of the document and are the same as the preset number;

[0024] In response to the number of titles being less than or equal to a preset number, it is determined that the first target identification range is all unstructured data and the second target identification range is all unstructured data.

[0025] Specifically, the semantic annotation of the unstructured data corresponding to the first target recognition range to obtain the second attribute information corresponding to the unstructured data includes:

[0026] Obtaining a preset synonym library and a second target recognition attribute;

[0027] According to the preset synonym library and the second target recognition attribute, the corresponding unstructured data within the first target recognition range is traversed and processed;

[0028] In response to the unstructured data corresponding to the first target recognition range intersecting with the preset synonym library and the second target recognition attribute, using the content corresponding to the intersection as the second attribute information;

[0029] The corresponding unstructured data within the first target recognition range and the content corresponding to the non-intersection between the preset synonym library and the second target recognition attribute are taken as target content, and the occurrence frequency of each target content is counted;

[0030] The target content corresponding to the occurrence frequency greater than the preset frequency is used as the second attribute information.

[0031] Specifically, the training process of the sentence attribute discrimination model includes:

[0032] Acquire an initial sentence and initial third attribute information corresponding to the initial sentence;

[0033] Establishing an attribute discrimination data set according to the initial sentence and the initial third attribute information;

[0034] Dividing the attribute discrimination data set into a training data set and a test data set;

[0035] The initial sentence attribute discrimination model is trained using the data in the training data set to obtain a sentence attribute discrimination model.

[0036] Specifically, the method further includes:

[0037] Acquire data in a test data set, wherein each data in the test data set includes an initial sentence and initial third attribute information corresponding to the initial sentence;

[0038] For each data in the test data set, the initial sentence is input into a sentence attribute discrimination model, and the training third attribute information is obtained through processing by the sentence attribute discrimination model;

[0039] Calculating the evaluation accuracy of the sentence attribute discrimination model based on the initial third attribute information and the training third attribute information;

[0040] It is determined that the evaluation accuracy is higher than a preset accuracy threshold, and the sentence attribute discrimination model training is completed.

[0041] Based on the same inventive concept, the second aspect of the present disclosure proposes a device for labeling unstructured data, comprising:

[0042] A title recognition module is configured to obtain unstructured data, recognize the unstructured data, and obtain at least one title;

[0043] A first attribute information determination module is configured to perform semantic annotation on all titles to obtain first attribute information corresponding to the unstructured data;

[0044] An identification range determination module is configured to obtain the number of titles and determine a first target identification range and a second target identification range according to the number of titles;

[0045] A second attribute information determination module is configured to perform semantic annotation on the corresponding unstructured data within the first target recognition range to obtain second attribute information corresponding to the unstructured data;

[0046] A splitting module is configured to perform sentence splitting on the corresponding unstructured data within the second target recognition range to obtain at least one sentence;

[0047] A third attribute information determination module is configured to input the at least one sentence into a pre-trained sentence attribute discrimination model, and obtain third attribute information corresponding to the unstructured data through processing by the sentence attribute discrimination model;

[0048] The storage module is configured to store the unstructured data, the first attribute information, the second attribute information and the third attribute information in a database accordingly.

[0049] Specifically, the title identification module is configured to:

[0050] Traversing the unstructured data, and taking characters in the unstructured data that meet a preset format as titles;

[0051] Obtaining all paragraphs contained in the unstructured data, and determining the length of characters contained in each paragraph;

[0052] In response to the existence of a target paragraph whose character length is less than or equal to a preset length threshold, the target paragraph is used as a title.

[0053] Specifically, the first attribute information determination module is configured to:

[0054] Obtain a preset synonym library, and compare each title in all titles with the preset synonym library;

[0055] For each title: in response to the title and the preset synonym library having an intersection, determining the content corresponding to the intersection as the initial first attribute information; obtaining a first target recognition attribute, determining a first word vector corresponding to the title and a second word vector corresponding to the first target recognition attribute, and calculating the similarity between the first word vector and the second word vector; in response to the similarity being greater than a similarity threshold, determining the content corresponding to the title as the initial first attribute information;

[0056] The initial first attribute information corresponding to all titles is counted to obtain the first attribute information corresponding to the unstructured data.

[0057] Specifically, the identification range determination module is configured to:

[0058] Get the number of titles;

[0059] In response to the number of titles being greater than a preset number, determining that a first target identification range is from the beginning of the document of the unstructured data to a target title, and a second target identification range is from the target title to the end of the document of the unstructured data, wherein the target titles are titles that are in sequence starting from the beginning of the document and are the same as the preset number;

[0060] In response to the number of titles being less than or equal to a preset number, it is determined that the first target identification range is all unstructured data and the second target identification range is all unstructured data.

[0061] Specifically, the second attribute information determination module is specifically configured as follows:

[0062] Obtaining a preset synonym library and a second target recognition attribute;

[0063] According to the preset synonym library and the second target recognition attribute, the corresponding unstructured data within the first target recognition range is traversed and processed;

[0064] In response to the unstructured data corresponding to the first target recognition range intersecting with the preset synonym library and the second target recognition attribute, using the content corresponding to the intersection as the second attribute information;

[0065] The corresponding unstructured data within the first target recognition range and the content corresponding to the non-intersection between the preset synonym library and the second target recognition attribute are taken as target content, and the occurrence frequency of each target content is counted;

[0066] The target content corresponding to the occurrence frequency greater than the preset frequency is used as the second attribute information.

[0067] Specifically, the device further includes a model training module, and the model training module is specifically configured as follows:

[0068] Acquire an initial sentence and initial third attribute information corresponding to the initial sentence;

[0069] Establishing an attribute discrimination data set according to the initial sentence and the initial third attribute information;

[0070] Dividing the attribute discrimination data set into a training data set and a test data set;

[0071] The initial sentence attribute discrimination model is trained using the data in the training data set to obtain a sentence attribute discrimination model.

[0072] Specifically, the model training module is further configured as follows:

[0073] Acquire data in a test data set, wherein each data in the test data set includes an initial sentence and initial third attribute information corresponding to the initial sentence;

[0074] For each data in the test data set, the initial sentence is input into a sentence attribute discrimination model, and the training third attribute information is obtained through processing by the sentence attribute discrimination model;

[0075] Calculating the evaluation accuracy of the sentence attribute discrimination model based on the initial third attribute information and the training third attribute information;

[0076] It is determined that the evaluation accuracy is higher than a preset accuracy threshold, and the sentence attribute discrimination model training is completed.

[0077] Based on the same inventive concept, the third aspect of the present disclosure proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the above-mentioned unstructured data labeling method when executing the computer program.

[0078] Based on the same inventive concept, the fourth aspect of the present disclosure proposes a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the unstructured data labeling method as described above.

[0079] As can be seen from the above, the present disclosure proposes a method, device, electronic device and storage medium for labeling unstructured data, which obtains unstructured data, identifies the unstructured data, obtains at least one title, and semantically annotates all titles to obtain the first attribute information corresponding to the unstructured data. Since the first attribute information is usually located at the title, the title of the unstructured data is identified, and the first attribute information located at the title is accurately identified, which reduces the recognition range and improves the recognition accuracy. The number of titles is obtained, and the first target recognition range and the second target recognition range are determined according to the number of titles. The corresponding unstructured data within the first target recognition range are semantically annotated to obtain the second attribute information corresponding to the unstructured data. The corresponding unstructured data within the second target recognition range is sentence-splitting to obtain at least one sentence, and the at least one sentence is input into a sentence attribute discrimination model obtained in advance. After being processed by the sentence attribute discrimination model, the third attribute information corresponding to the unstructured data is obtained, and the sentence attribute discrimination model is used to analyze and process the sentence, and the corresponding third attribute information is output, and the third attribute information is determined more accurately. The unstructured data, the first attribute information, the second attribute information and the third attribute information are stored in the database accordingly, which realizes the labeling of different levels of unstructured data, and has the characteristics of accuracy, speed and efficiency. The recognition and labeling results are stored in the database in an orderly manner, which realizes the transformation and integration of unstructured data into structured data. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0081] Figure 1 A flowchart of a method for labeling unstructured data according to an embodiment of the present disclosure;

[0082] Figure 2 A flowchart of paragraph recognition according to an embodiment of the present disclosure;

[0083] Figure 3 A structural block diagram of a device for labeling unstructured data according to an embodiment of the present disclosure;

[0084] Figure 4 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0085] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0086] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0087] The terms used in this disclosure are explained as follows:

[0088] DQN: Deep Q network (DQN) is an algorithm used to solve reinforcement learning problems.

[0089] DDPG: DDPG (Deep Deterministic Policy Gradient) is an off-policy algorithm and can only be used in continuous action space.

[0090] PPO: Proximal Policy Optimization (PPO) algorithm.

[0091] TRPO: Trust region policy optimization (TRPO).

[0092] A3C: Reinforcement learning algorithm (Asynchronous Advantage Actor-Critic, A3C).

[0093] SAC: SAC is a maximum entropy-based policy learning method (Soft Actor-Critic, SAC), which improves learning speed and stability by adding entropy terms and randomization strategies.

[0094] With the rapid development of the economy and society, scientific research has become increasingly extensive and in-depth, accumulating a large amount of scientific research project data. These data include not only various research results, experimental data and technical literature, but also multi-dimensional information such as project management, funding and expert allocation.

[0095] However, a large amount of data is unstructured, and traditional data processing methods find it difficult to effectively extract and utilize this information.

[0096] Based on the above description, this embodiment proposes a method for labeling unstructured data, such as Figure 1 As shown, the method includes:

[0097] Step 101: Obtain unstructured data, identify the unstructured data, and obtain at least one title.

[0098] In the specific implementation, unstructured data is obtained, which is data with irregular or incomplete data structure and no predefined data, that is, data that is not convenient to be represented by a two-dimensional logical table of a database. It includes office documents of all formats, text, pictures, HTML, various reports, images, and audio / video information, etc.

[0099] In this embodiment, the unstructured data is preferably unstructured data of scientific and technological projects.

[0100] Unstructured data is identified and processed to obtain at least one title contained in the unstructured data. In the unstructured data structure of scientific and technological projects, there are usually titles. Although these titles may not completely match the target annotation attributes, by making full use of these title information, some data information can be filtered out, and the word scale of the unstructured data of scientific and technological projects can be reduced, and the content information can be located and marked, which will greatly reduce the difficulty of subsequent annotation processing and improve the annotation efficiency of unstructured data of scientific and technological projects.

[0101] Step 102: semantically annotate all titles to obtain first attribute information corresponding to the unstructured data.

[0102] In a specific implementation, semantic annotation is performed on each of at least one identified title to obtain first attribute information corresponding to the unstructured data, wherein the first attribute information is attribute information contained in the title in the unstructured data, including at least one of the following: research field, research objective, research program, research plan, assessment indicators, results, demonstration, technology promotion, abstract, funding / budget and experts and (project) members / participants.

[0103] Step 103, obtaining the number of titles, and determining the first target recognition range and the second target recognition range according to the number of titles.

[0104] In a specific implementation, the number of recognized titles is counted, and a first target recognition range and a second target recognition range are determined based on the number of titles, wherein the first target recognition range and the second target recognition range are both ranges included in the unstructured data, and the first target recognition range and the second target recognition range may be the same or different.

[0105] Step 104: semantically annotate the corresponding unstructured data within the first target recognition range to obtain second attribute information corresponding to the unstructured data.

[0106] During specific implementation, semantic annotation is performed on the corresponding unstructured data within the first target recognition range to obtain second attribute information corresponding to the unstructured data, wherein the second attribute information is the attribute information contained in the title of the unstructured data, including at least one of the following: project name, project number, person in charge, unit and time of the scientific and technological project.

[0107] Step 105: perform sentence splitting on the corresponding unstructured data within the second target recognition range to obtain at least one sentence.

[0108] In a specific implementation, the unstructured data within the second target recognition range is subjected to sentence splitting, that is, the corresponding unstructured data within the second target recognition range is split from paragraphs into sentences to obtain at least one sentence.

[0109] Step 106: input the at least one sentence into a pre-trained sentence attribute discrimination model, and obtain third attribute information corresponding to the unstructured data through processing by the sentence attribute discrimination model.

[0110] In the specific implementation, the initial sentence attribute discrimination model is trained in advance, and the sentence attribute discrimination model is obtained after the training is completed, wherein the initial sentence attribute discrimination model is a model using a neural network structure, specifically a classification model based on machine learning, which aims to analyze the input sentence to determine its specific annotation attributes in the scientific and technological project. The input of the model is a sentence, and the model can automatically identify the attributes corresponding to the sentence after being trained with historical scientific and technological project data. The third attribute information includes at least one of the following: DQN model, DDPG model, PPO model, TRPO model, A3C model, SAC model.

[0111] The at least one sentence is input into a pre-trained sentence attribute discrimination model, and processed by the sentence attribute discrimination model to obtain third attribute information corresponding to the unstructured data, wherein the third attribute information is attribute information contained in the title of the unstructured data, including at least one of the following: research field, research objective, research content, research program, research plan, assessment indicator, achievement, demonstration, and technology promotion.

[0112] Step 107: store the unstructured data, the first attribute information, the second attribute information, and the third attribute information in a database accordingly.

[0113] In specific implementation, after obtaining the first attribute information, the second attribute information and the third attribute information, the unstructured data, the first attribute information, the second attribute information and the third attribute information are stored correspondingly in a database, so that the unstructured data containing the attribute information can be subsequently searched from the database according to the attribute information.

[0114] In this embodiment, it is preferred to store in the form of key-value pairs, that is, a data structure in the form of "type-value" is used and stored in the database accordingly.

[0115] Through the above scheme, unstructured data is obtained, the unstructured data is identified, at least one title is obtained, all titles are semantically annotated, and the first attribute information corresponding to the unstructured data is obtained. Because the first attribute information is usually located at the title, the title of the unstructured data is identified, and the first attribute information located at the title is accurately identified, which reduces the recognition range and improves the recognition accuracy. The number of titles is obtained, and the first target recognition range and the second target recognition range are determined according to the number of titles. The unstructured data corresponding to the first target recognition range is semantically annotated to obtain the second attribute information corresponding to the unstructured data. The unstructured data corresponding to the second target recognition range is sentence-splitting to obtain at least one sentence, and the at least one sentence is input into the pre-trained sentence attribute discrimination model. After being processed by the sentence attribute discrimination model, the third attribute information corresponding to the unstructured data is obtained. The sentence attribute discrimination model is used to analyze and process the sentence, and the corresponding third attribute information is output. The third attribute information is determined more accurately. The unstructured data, the first attribute information, the second attribute information and the third attribute information are stored in the database accordingly, which realizes the labeling of different levels of unstructured data, and has the characteristics of accuracy, speed and efficiency. The recognition and labeling results are stored in the database in an orderly manner, which realizes the transformation and integration of unstructured data into structured data.

[0116] In some embodiments, after obtaining the unstructured data, the unstructured data may be preprocessed to convert the unstructured data into a data format recognizable by the program.

[0117] In this embodiment, the unstructured data of the scientific and technological project refers to the Word, PDF, PPT, and TXT files stored as attachments in the scientific and technological project. These files can be converted into program-recognizable data with the help of installation packages such as python-word, PyPDF2, python-pptx, and pandas.

[0118] Format conversion refers to the recognition and processing of images and table data in the unstructured data of scientific and technological projects, and the conversion of image and table data into text data for content semantic analysis and annotation.

[0119] Through the above scheme, some important information in the unstructured data of scientific and technological projects, such as funds / budgets, (project) members / participants, experts, etc., is stored in pictures and tables, so the annotation of picture and table data information is an indispensable part of the annotation of unstructured data of scientific and technological projects. In order to facilitate the understanding of the data content and meaning in pictures and tables, pictures are converted into text data through optical character recognition technology (OCR), and tables are also converted into text data with the help of text data recognition, processing and annotation for subsequent text data recognition, processing and annotation.

[0120] In some embodiments, step 101 specifically includes:

[0121] Step 1011, traversing the unstructured data, and taking characters in the unstructured data that meet a preset format as titles;

[0122] Step 1012, obtaining all paragraphs contained in the unstructured data, and determining the length of characters contained in each paragraph;

[0123] Step 1013: In response to the existence of a target paragraph whose character length is less than or equal to a preset length threshold, the target paragraph is used as a title.

[0124] In specific implementation, in the unstructured data structure of scientific and technological projects, the title generally exists in a unique paragraph format. The unstructured data is traversed and processed, and if there are characters that meet the preset format, the characters that meet the preset format are used as the title.

[0125] For example, taking a Word file as an example, the python-word installation package is used to detect whether the style name of each paragraph contains the "Heading" character, and those containing the "Heading" character are identified as titles.

[0126] All paragraphs contained in the unstructured data are obtained, and the character length of each paragraph is determined. If there is a target paragraph whose character length is less than or equal to a preset length threshold, the target paragraph is used as a title.

[0127] Exemplarily, the title in a scientific and technological project document is usually 2 to 4 words, and no longer than 8 to 9 words. The preset length threshold is 10. If the length of the characters contained in a paragraph is not more than 10 characters, the paragraph is determined to be a title.

[0128] In this embodiment, the title recognition flow chart is as follows: Figure 2 As shown, the unstructured data is retrieved in paragraph order to detect whether the paragraph meets the title format. If so, the paragraph is a title. If not, it is determined whether the length of the paragraph words meets the standard length of the title. If so, the paragraph is determined to be a title. If not, the paragraph is not a title.

[0129] In order to improve the efficiency of document title detection, for paragraphs that have been determined to be document titles through paragraph format identification, title detection based on paragraph word length will no longer be performed.

[0130] Through the above scheme, by identifying the title according to the preset format and preset length, the identified title is more comprehensive and accurate.

[0131] In some embodiments, based on title detection, the unstructured data of scientific and technological projects can be divided into several blocks for further processing and identification. After title detection, the unstructured data of scientific and technological projects is reduced from large to small, and the word scale corresponding to the text unit to be processed becomes smaller, which will reduce the difficulty of text analysis and processing. At the same time, based on title detection, the unstructured data of scientific and technological projects can be marked for content location.

[0132] In some embodiments, step 102 specifically includes:

[0133] Step 1021, obtain a preset synonym library, and compare each title in all titles with the preset synonym library.

[0134] Step 1022, for each title: in response to the existence of an intersection between the title and the preset synonym library, determine that the content corresponding to the intersection is the initial first attribute information; obtain the first target recognition attribute, determine the first word vector corresponding to the title and the second word vector corresponding to the first target recognition attribute, and calculate the similarity between the first word vector and the second word vector; in response to the similarity being greater than a similarity threshold, determine that the content corresponding to the title is the initial first attribute information.

[0135] Step 1023, counting the initial first attribute information corresponding to all titles to obtain the first attribute information corresponding to the unstructured data.

[0136] In a specific implementation, a preset synonym library is obtained, wherein the preset synonym library contains all synonyms corresponding to the first attribute information. Each of all the recognized titles is compared with the preset synonym library.

[0137] For each title:

[0138] If the title and the preset synonym library have an intersection, it means that there are synonyms with the same meaning as the title in the preset synonym library. At this time, the content corresponding to the intersection is the initial first attribute information, that is, the content in the title corresponding to the synonym is the initial first attribute information.

[0139] The words corresponding to the title are converted into a first word vector by word embedding, and the words corresponding to the first target identification attribute are converted into a second word vector by word embedding, wherein the first target identification attribute includes at least one of the following: research field, research objective, research program, research plan, assessment index, achievement, demonstration, technology promotion, abstract, funding / budget and experts and (project) members / participants.

[0140] Determine a first word vector corresponding to the title and a second word vector corresponding to the first target recognition attribute, and calculate the similarity between the first word vector and the second word vector; in response to the similarity being greater than a similarity threshold, determine that the content corresponding to the title is initial first attribute information.

[0141] Generally speaking, if the similarity is greater than 0.7, it can be determined that they are different expressions of the same meaning. However, in this embodiment, because the title of the scientific and technological project document is detected, a higher accuracy is required, so the similarity judgment standard is set to 0.85, which has a better effect in application.

[0142] The initial first attribute information corresponding to all titles is counted to obtain the first attribute information corresponding to the unstructured data.

[0143] Through the above scheme, the initial first attribute information is screened by using a synonym library comparison and a similarity-based word embedding method, which makes the judgment of the initial first attribute information more accurate, and thus the first attribute information corresponding to the unstructured data obtained by statistics is more accurate.

[0144] In this embodiment, paragraphs with titles such as research field, research goal, research program, research plan, assessment index, achievement, demonstration, technology promotion, abstract, funding / budget, experts and (project) members / participants and similar expressions have their own uniqueness, which can be marked according to the sequence number of the paragraph, and these paragraphs will no longer be analyzed and processed later. For paragraphs with titles such as research content and similar expressions, there are cases where they contain other target annotation attribute content, which requires further detailed analysis and identification. For other titles that cannot be identified as similar to the target annotation attribute, subsequent analysis, identification and identification are required.

[0145] In some embodiments, step 103 specifically includes:

[0146] Step 1031, obtaining the number of titles;

[0147] Step 1032, in response to the number of titles being greater than a preset number, determining that a first target recognition range is from the beginning of the document of the unstructured data to a target title, and a second target recognition range is from a target title to the end of the document of the unstructured data, wherein the target titles are titles that are in sequence starting from the beginning of the document and are the same as the preset number;

[0148] Step 1033, in response to the number of titles being less than or equal to a preset number, determining that the first target recognition range is all unstructured data and the second target recognition range is all unstructured data.

[0149] In specific implementation, the number of recognized titles is counted. If the number of titles is greater than a preset number, the first target recognition range is determined to be from the beginning of the document of the unstructured data to the target title, and the second target recognition range is determined to be from the target title to the end of the document of the unstructured data, wherein the target title is a title starting from the beginning of the document and having the same value as the preset number.

[0150] Exemplarily, the preset number is 3. If the number of recognized titles is 5, then the first target recognition range is from the beginning of the document of the unstructured data to the third title, and the second target recognition range is from the third title to the end of the document of the unstructured data.

[0151] If the number of titles is less than or equal to the preset number, it is determined that the first target recognition range is all unstructured data and the second target recognition range is all unstructured data.

[0152] Through the above scheme, the recognition range is determined according to the number of titles. In scientific and technological project documents, the project name, project number, person in charge, unit and time are usually at the front of the document. If the number of titles is large, the recognition range can be appropriately narrowed to improve the recognition efficiency.

[0153] In some embodiments, step 104 specifically includes:

[0154] Step 1041, obtaining a preset synonym library and a second target recognition attribute;

[0155] Step 1042, traversing and processing the corresponding unstructured data within the first target recognition range according to the preset synonym library and the second target recognition attribute;

[0156] Step 1043, in response to the corresponding unstructured data within the first target recognition range intersecting with the preset synonym library and the second target recognition attribute, using the content corresponding to the intersection as second attribute information;

[0157] Step 1044, taking the corresponding unstructured data within the first target recognition range and the content corresponding to the preset synonym library and the second target recognition attribute that do not have an intersection as target content, and counting the occurrence frequency of each target content;

[0158] Step 1045: taking the target content corresponding to the occurrence frequency greater than the preset frequency as the second attribute information.

[0159] In a specific implementation, a preset synonym library and a second target identification attribute are obtained, wherein the second target identification attribute includes at least one of the following: the project name, project number, person in charge, unit and time of the scientific and technological project.

[0160] The corresponding unstructured data within the first target recognition range is traversed and processed according to the preset synonym library and the second target recognition attribute. If the corresponding unstructured data within the first target recognition range has an intersection with the preset synonym library and the second target recognition attribute, the content corresponding to the intersection is used as the second attribute information.

[0161] The unstructured data corresponding to the first target recognition range and the content corresponding to the non-intersection between the preset synonym library and the second target recognition attribute is taken as the target content, that is, the target content is the content that does not have an intersection with the second target recognition attribute in the preset synonym library.

[0162] The occurrence frequency of each target content is counted, and the target content corresponding to the occurrence frequency greater than the preset frequency is used as the second attribute information.

[0163] Through the above scheme, the second attribute information is identified based on the synonym library and the frequency of occurrence, ensuring the effective identification and extraction of key data. By using the frequency of occurrence for identification again after using the synonym library for identification, the accuracy and comprehensiveness of information extraction can be effectively improved, especially when facing complex texts and ambiguous expressions, the recognition accuracy of key information is improved.

[0164] In some embodiments, before traversing and processing the corresponding unstructured data within the first target identification scope according to the preset synonym library and the second target identification attribute, semantic analysis can be performed on the corresponding unstructured data within the first target identification scope to eliminate interference from invalid information such as project statements and letters of commitment.

[0165] In the unstructured data of scientific and technological projects, the content that has not been attributed usually has complex semantic connotations, and the length of words may be long. In this case, the analysis and processing method based on paragraphs and words is no longer applicable. Therefore, in some embodiments, a sentence attribute discrimination model is introduced to identify the third attribute information. The training process of the sentence attribute discrimination model includes:

[0166] Step 10A, obtaining an initial sentence and initial third attribute information corresponding to the initial sentence;

[0167] Step 10B, establishing an attribute discrimination data set according to the initial sentence and the initial third attribute information;

[0168] Step 10C, dividing the attribute discrimination data set into a training data set and a test data set;

[0169] Step 10D: Use the data in the training data set to train the initial sentence attribute discrimination model to obtain a sentence attribute discrimination model.

[0170] In a specific implementation, an initial sentence and initial third attribute information corresponding to the initial sentence are obtained, and an attribute discrimination data set is established based on the obtained initial sentence and the initial third attribute information. The attribute discrimination data set is divided to obtain a training data set and a test data set, wherein the training data set is used to train the initial sentence attribute discrimination model, and the test data set is used to test the trained model to determine whether the training is completed.

[0171] Exemplarily, the data in the attribute discrimination data set are randomly shuttled, and 80% are selected as the training data set and 20% as the test data set.

[0172] In another example, the data in the attribute discrimination data set are randomly shuffled, and 70% are selected as the training data set and 30% as the test data set.

[0173] The initial sentence attribute discrimination model is trained using the data in the training data set until a preset training end condition is met, thereby obtaining a sentence attribute discrimination model.

[0174] The preset training end condition includes at least one of the following: determining that all training data sets are input into the initial sentence attribute discrimination model for training, determining that the loss function of the initial sentence attribute discrimination model converges to a preset convergence threshold, or determining that the initial sentence attribute discrimination model is iteratively trained to a preset number of iterations.

[0175] Exemplarily, the preset training end condition is to determine that all training data sets are input into the initial sentence attribute discrimination model for training:

[0176] There are ten groups of data in the training data set, each group of data includes an initial sentence and initial third attribute information corresponding to the initial sentence. The preset training end condition is to determine that all data in the training data set are input into the initial sentence attribute discrimination model for training, that is, when the ten groups of data are input into the initial sentence attribute discrimination model, there is no data in the training data set that has not been input into the initial sentence attribute discrimination model, and the initial sentence attribute discrimination model training is determined to be completed, and a sentence attribute discrimination model is obtained.

[0177] In another example, the preset training end condition is to determine that the loss function of the initial sentence attribute discrimination model converges to a preset convergence threshold:

[0178] The training data set is input into the initial sentence attribute discrimination model for training, and the training result is output. A loss function is determined according to the training result and the actual output torque. The type of the loss function includes at least one of the following: mean square error loss function, cross entropy loss function, logarithmic loss function, exponential loss function, square loss function or absolute value loss function, etc. When the loss function converges to a preset convergence threshold, it is determined that the preset training end condition is met, and a sentence attribute discrimination model is obtained.

[0179] In another example, the preset training end condition is to determine that the initial sentence attribute discrimination model is iteratively trained to a preset number of iterations.

[0180] The training data set is input into the initial sentence attribute discrimination model for iterative training, the number of iterations is recorded, and when it is determined that the number of iterations is equal to the preset number of iterations, the preset training end condition is met, and the sentence attribute discrimination model is obtained.

[0181] In some embodiments, the method further comprises:

[0182] Step A, obtaining data in a test data set, wherein each data in the test data set includes an initial sentence and initial third attribute information corresponding to the initial sentence;

[0183] Step B, for each data in the test data set, inputting the initial sentence into a sentence attribute discrimination model, and obtaining training third attribute information through processing by the sentence attribute discrimination model;

[0184] Step C, calculating the evaluation accuracy of the sentence attribute discrimination model based on the initial third attribute information and the training third attribute information;

[0185] Step D: Determine that the evaluation accuracy is higher than a preset accuracy threshold, and the sentence attribute discrimination model training is completed.

[0186] In a specific implementation, data in a test data set in an attribute discrimination data set is obtained, and each data in the test data set includes an initial sentence and initial third attribute information corresponding to the initial sentence. The initial sentence in the data in the test data set is input into a sentence attribute discrimination model to obtain the corresponding initial third attribute information.

[0187] The first emotion information corresponding to each data in the test data set is compared with the initial emotion information to determine whether the first emotion information is the same as the initial emotion information. The evaluation accuracy of the building emotion evaluation model is calculated. When the evaluation accuracy is higher than the preset accuracy threshold, the sentence attribute discrimination model training is completed.

[0188] Exemplarily, the test data set contains 10 data. The initial third attribute information corresponding to each data in the test data set is compared with the training third attribute information respectively, and it is determined that among the 10 data, the number of the initial third attribute information and the training third attribute information is the same as 7, and the number of the initial third attribute information and the training third attribute information is different from 3, then the evaluation accuracy of the sentence attribute discrimination model is 70%.

[0189] The preset accuracy threshold is 60%, and the evaluation accuracy is greater than the preset accuracy threshold, and it is determined that the training of the sentence attribute discrimination model is completed.

[0190] Through the above scheme, based on the labeled attribute sentence data, a sentence attribute discrimination model is created and trained, and then the sentence attribute discrimination model is used to achieve the final annotation of the unstructured data of scientific and technological projects. Using sentences as analysis and annotation units can achieve accurate annotation of unstructured data of scientific and technological projects. When labeled as "research content", the corresponding sentence is "research content" without further subdivision. Sentence annotation using the sentence attribute discrimination model can completely replace paragraph annotation. However, the efficiency of sentence annotation is relatively low, so paragraph annotation still has its necessity.

[0191] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0192] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0193] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a device for labeling unstructured data.

[0194] refer to Figure 3 , Figure 3 The unstructured data labeling device of the embodiment includes:

[0195] The title recognition module 201 is configured to obtain unstructured data, recognize the unstructured data, and obtain at least one title;

[0196] A first attribute information determination module 202 is configured to perform semantic annotation on all titles to obtain first attribute information corresponding to the unstructured data;

[0197] The recognition range determination module 203 is configured to obtain the number of titles and determine the first target recognition range and the second target recognition range according to the number of titles;

[0198] The second attribute information determination module 204 is configured to perform semantic annotation on the corresponding unstructured data within the first target recognition range to obtain second attribute information corresponding to the unstructured data;

[0199] The splitting module 205 is configured to perform sentence splitting on the corresponding unstructured data within the second target recognition range to obtain at least one sentence;

[0200] The third attribute information determination module 206 is configured to input the at least one sentence into a pre-trained sentence attribute discrimination model, and obtain the third attribute information corresponding to the unstructured data through processing by the sentence attribute discrimination model;

[0201] The storage module 207 is configured to store the unstructured data, the first attribute information, the second attribute information and the third attribute information in a database accordingly.

[0202] In some embodiments, the title identification module 201 is configured to:

[0203] Traversing the unstructured data, and taking characters in the unstructured data that meet a preset format as titles;

[0204] Obtaining all paragraphs contained in the unstructured data, and determining the length of characters contained in each paragraph;

[0205] In response to the existence of a target paragraph whose character length is less than or equal to a preset length threshold, the target paragraph is used as a title.

[0206] In some embodiments, the first attribute information determining module 202 is configured to:

[0207] Obtain a preset synonym library, and compare each title in all titles with the preset synonym library;

[0208] For each title: in response to the title and the preset synonym library having an intersection, determining the content corresponding to the intersection as the initial first attribute information; obtaining a first target recognition attribute, determining a first word vector corresponding to the title and a second word vector corresponding to the first target recognition attribute, and calculating the similarity between the first word vector and the second word vector; in response to the similarity being greater than a similarity threshold, determining the content corresponding to the title as the initial first attribute information;

[0209] The initial first attribute information corresponding to all titles is counted to obtain the first attribute information corresponding to the unstructured data.

[0210] In some embodiments, the recognition range determination module 203 is configured to:

[0211] Get the number of titles;

[0212] In response to the number of titles being greater than a preset number, determining that a first target identification range is from the beginning of the document of the unstructured data to a target title, and a second target identification range is from the target title to the end of the document of the unstructured data, wherein the target titles are titles that are in sequence starting from the beginning of the document and are the same as the preset number;

[0213] In response to the number of titles being less than or equal to a preset number, it is determined that the first target identification range is all unstructured data and the second target identification range is all unstructured data.

[0214] In some embodiments, the second attribute information determination module 204 is specifically configured to:

[0215] Obtaining a preset synonym library and a second target recognition attribute;

[0216] According to the preset synonym library and the second target recognition attribute, the corresponding unstructured data within the first target recognition range is traversed and processed;

[0217] In response to the unstructured data corresponding to the first target recognition range intersecting with the preset synonym library and the second target recognition attribute, using the content corresponding to the intersection as the second attribute information;

[0218] The corresponding unstructured data within the first target recognition range and the content corresponding to the non-intersection between the preset synonym library and the second target recognition attribute are taken as target content, and the occurrence frequency of each target content is counted;

[0219] The target content corresponding to the occurrence frequency greater than the preset frequency is used as the second attribute information.

[0220] In some embodiments, the device further includes a model training module, and the model training module is specifically configured to:

[0221] Acquire an initial sentence and initial third attribute information corresponding to the initial sentence;

[0222] Establishing an attribute discrimination data set according to the initial sentence and the initial third attribute information;

[0223] Dividing the attribute discrimination data set into a training data set and a test data set;

[0224] The initial sentence attribute discrimination model is trained using the data in the training data set to obtain a sentence attribute discrimination model.

[0225] In some embodiments, the model training module is further configured to:

[0226] Acquire data in a test data set, wherein each data in the test data set includes an initial sentence and initial third attribute information corresponding to the initial sentence;

[0227] For each data in the test data set, the initial sentence is input into a sentence attribute discrimination model, and the training third attribute information is obtained through processing by the sentence attribute discrimination model;

[0228] Calculating the evaluation accuracy of the sentence attribute discrimination model based on the initial third attribute information and the training third attribute information;

[0229] It is determined that the evaluation accuracy is higher than a preset accuracy threshold, and the sentence attribute discrimination model training is completed.

[0230] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0231] The device of the above embodiment is used to implement the corresponding unstructured data labeling method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0232] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the unstructured data labeling method described in any of the above embodiments is implemented.

[0233] Figure 4 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0234] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0235] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0236] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0237] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0238] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0239] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0240] The electronic device of the above embodiment is used to implement the corresponding unstructured data labeling method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0241] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the unstructured data labeling method described in any of the above embodiments.

[0242] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0243] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the unstructured data labeling method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0244] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0245] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0246] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0247] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0248] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0249] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the known power / ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0250] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0251] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A method for labeling unstructured data, characterized in that: include: Acquire unstructured data, identify the unstructured data, and obtain at least one title; Performing semantic annotation on all titles to obtain first attribute information corresponding to the unstructured data; Acquire the number of titles, and determine the first target recognition range and the second target recognition range according to the number of titles; Performing semantic annotation on the corresponding unstructured data within the first target recognition range to obtain second attribute information corresponding to the unstructured data; Perform sentence splitting on the corresponding unstructured data within the second target recognition range to obtain at least one sentence; Inputting the at least one sentence into a pre-trained sentence attribute discrimination model, and processing the sentence attribute discrimination model to obtain third attribute information corresponding to the unstructured data; The unstructured data, the first attribute information, the second attribute information and the third attribute information are correspondingly stored in a database.

2. The method according to claim 1, characterized in that The step of identifying the unstructured data to obtain at least one title includes: Traversing the unstructured data, and taking characters in the unstructured data that meet a preset format as titles; Obtaining all paragraphs contained in the unstructured data, and determining the length of characters contained in each paragraph; In response to the existence of a target paragraph whose character length is less than or equal to a preset length threshold, the target paragraph is used as a title.

3. The method according to claim 1, characterized in that The semantically annotating all titles to obtain first attribute information corresponding to the unstructured data includes: Obtain a preset synonym library, and compare each title in all titles with the preset synonym library; For each title: in response to the title and the preset synonym library having an intersection, determining the content corresponding to the intersection as the initial first attribute information; obtaining a first target recognition attribute, determining a first word vector corresponding to the title and a second word vector corresponding to the first target recognition attribute, and calculating the similarity between the first word vector and the second word vector; in response to the similarity being greater than a similarity threshold, determining the content corresponding to the title as the initial first attribute information; The initial first attribute information corresponding to all titles is counted to obtain the first attribute information corresponding to the unstructured data.

4. The method according to claim 1, characterized in that: The obtaining of the number of titles and determining the first target recognition range and the second target recognition range according to the number of titles include: Get the number of titles; In response to the number of titles being greater than a preset number, determining that a first target identification range is from the beginning of the document of the unstructured data to a target title, and a second target identification range is from the target title to the end of the document of the unstructured data, wherein the target titles are titles that are in sequence starting from the beginning of the document and are the same as the preset number; In response to the number of titles being less than or equal to a preset number, it is determined that the first target identification range is all unstructured data and the second target identification range is all unstructured data.

5. The method according to claim 1, characterized in that The semantically annotating the unstructured data corresponding to the first target recognition range to obtain second attribute information corresponding to the unstructured data includes: Obtaining a preset synonym library and a second target recognition attribute; According to the preset synonym library and the second target recognition attribute, the corresponding unstructured data within the first target recognition range is traversed and processed; In response to the unstructured data corresponding to the first target recognition range intersecting with the preset synonym library and the second target recognition attribute, using the content corresponding to the intersection as the second attribute information; The corresponding unstructured data within the first target recognition range and the content corresponding to the non-intersection between the preset synonym library and the second target recognition attribute are taken as target content, and the occurrence frequency of each target content is counted; The target content corresponding to the occurrence frequency greater than the preset frequency is used as the second attribute information.

6. The method according to claim 1, characterized in that The training process of the sentence attribute discrimination model includes: Acquire an initial sentence and initial third attribute information corresponding to the initial sentence; Establishing an attribute discrimination data set according to the initial sentence and the initial third attribute information; Dividing the attribute discrimination data set into a training data set and a test data set; The initial sentence attribute discrimination model is trained using the data in the training data set to obtain a sentence attribute discrimination model.

7. The method according to claim 6, characterized in that Also includes: Acquire data in a test data set, wherein each data in the test data set includes an initial sentence and initial third attribute information corresponding to the initial sentence; For each data in the test data set, the initial sentence is input into a sentence attribute discrimination model, and the training third attribute information is obtained through processing by the sentence attribute discrimination model; Calculating the evaluation accuracy of the sentence attribute discrimination model based on the initial third attribute information and the training third attribute information; It is determined that the evaluation accuracy is higher than a preset accuracy threshold, and the sentence attribute discrimination model training is completed.

8. A device for labeling unstructured data, characterized in that: include: A title recognition module is configured to obtain unstructured data, recognize the unstructured data, and obtain at least one title; A first attribute information determination module is configured to perform semantic annotation on all titles to obtain first attribute information corresponding to the unstructured data; An identification range determination module is configured to obtain the number of titles and determine a first target identification range and a second target identification range according to the number of titles; A second attribute information determination module is configured to perform semantic annotation on the corresponding unstructured data within the first target recognition range to obtain second attribute information corresponding to the unstructured data; A splitting module is configured to perform sentence splitting on the corresponding unstructured data within the second target recognition range to obtain at least one sentence; A third attribute information determination module is configured to input the at least one sentence into a pre-trained sentence attribute discrimination model, and obtain third attribute information corresponding to the unstructured data through processing by the sentence attribute discrimination model; The storage module is configured to store the unstructured data, the first attribute information, the second attribute information and the third attribute information in a database accordingly.

9. The device according to claim 8, characterized in that The title identification module is configured to: Traversing the unstructured data, and taking characters in the unstructured data that meet a preset format as titles; Obtaining all paragraphs contained in the unstructured data, and determining the length of characters contained in each paragraph; In response to the existence of a target paragraph whose character length is less than or equal to a preset length threshold, the target paragraph is used as a title.

10. The device according to claim 8, characterized in that The first attribute information determination module is configured to: Obtain a preset synonym library, and compare each title in all titles with the preset synonym library; For each title: in response to the title and the preset synonym library having an intersection, determining the content corresponding to the intersection as the initial first attribute information; obtaining a first target recognition attribute, determining a first word vector corresponding to the title and a second word vector corresponding to the first target recognition attribute, and calculating the similarity between the first word vector and the second word vector; In response to the similarity being greater than a similarity threshold, determining that the content corresponding to the title is initial first attribute information; The initial first attribute information corresponding to all titles is counted to obtain the first attribute information corresponding to the unstructured data.

11. The device according to claim 8, characterized in that The identification range determination module is configured to: Get the number of titles; In response to the number of titles being greater than a preset number, determining that a first target identification range is from the beginning of the document of the unstructured data to a target title, and a second target identification range is from the target title to the end of the document of the unstructured data, wherein the target titles are titles that are in sequence starting from the beginning of the document and are the same as the preset number; In response to the number of titles being less than or equal to a preset number, it is determined that the first target identification range is all unstructured data and the second target identification range is all unstructured data.

12. The device according to claim 8, characterized in that The second attribute information determination module is specifically configured as follows: Obtaining a preset synonym library and a second target recognition attribute; According to the preset synonym library and the second target recognition attribute, the corresponding unstructured data within the first target recognition range is traversed and processed; In response to the unstructured data corresponding to the first target recognition range intersecting with the preset synonym library and the second target recognition attribute, using the content corresponding to the intersection as the second attribute information; The corresponding unstructured data within the first target recognition range and the content corresponding to the non-intersection between the preset synonym library and the second target recognition attribute are taken as target content, and the occurrence frequency of each target content is counted; The target content corresponding to the occurrence frequency greater than the preset frequency is used as the second attribute information.

13. The device according to claim 8, characterized in that The device also includes a model training module, which is specifically configured to: Acquire an initial sentence and initial third attribute information corresponding to the initial sentence; Establishing an attribute discrimination data set according to the initial sentence and the initial third attribute information; Dividing the attribute discrimination data set into a training data set and a test data set; The initial sentence attribute discrimination model is trained using the data in the training data set to obtain a sentence attribute discrimination model.

14. The device according to claim 13, characterized in that The model training module is further specifically configured as follows: Acquire data in a test data set, wherein each data in the test data set includes an initial sentence and initial third attribute information corresponding to the initial sentence; For each data in the test data set, the initial sentence is input into a sentence attribute discrimination model, and the training third attribute information is obtained through processing by the sentence attribute discrimination model; Calculating the evaluation accuracy of the sentence attribute discrimination model based on the initial third attribute information and the training third attribute information; It is determined that the evaluation accuracy is higher than a preset accuracy threshold, and the sentence attribute discrimination model training is completed.

15. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

16. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 7.

Images