Annotation method, annotation device, electronic device and storage medium

Abstract

The present application provides an annotation method, an annotation device, an electronic device and a storage medium, which belong to the field of artificial intelligence technology. The method includes: reading reference fields and reference annotations from a preset data warehouse; constructing an annotation dictionary based on the reference fields and reference annotations; the annotation dictionary includes key-value pairs, and the key-value pairs include one-to-one corresponding reference fields and reference annotations; reading the original field from the local library, and the annotation content of the original field is empty; loading the annotation dictionary with the original field as the database key to obtain the loading result; if the loading result is successful, extracting the reference annotation, and annotating the original field according to the reference annotation to obtain the target field; the target field includes field content and annotation content; if the loading result is a failure to load, performing format conversion on the original field to obtain an intermediate field, and completing the annotation of the intermediate field to obtain the target field. The present application can improve the efficiency of annotation.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an annotation method, an annotation device, an electronic device, and a storage medium. Background Art

[0002] At present, manual annotation is often used when annotating fields in data tables. This method often requires manual writing of script data to annotate the original fields that lack annotations. The workload is large and affects the annotation efficiency. Therefore, how to improve the annotation efficiency has become a technical problem that needs to be solved urgently. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose an annotation method, an annotation device, an electronic device and a storage medium, aiming to improve annotation efficiency.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application provides an annotation method, the method comprising:

[0005] Read reference fields and reference comments from a preset data warehouse;

[0006] Constructing an annotation dictionary based on the reference field and the reference annotation; wherein the annotation dictionary includes at least one key-value pair, and the key-value pair includes the reference field and the reference annotation in a one-to-one correspondence;

[0007] Reading an original field from a local library; wherein the annotation content of the original field is empty;

[0008] Loading the annotation dictionary using the original field as a database key to obtain a loading result;

[0009] If the loading result is successful, the reference annotation is extracted, and the original field is annotated according to the reference annotation to obtain a target field; wherein the target field includes field content and annotation content, the field content is derived from the original field, and the annotation content is derived from the reference annotation;

[0010] If the loading result is a loading failure, the original field is format-converted to obtain an intermediate field, and the intermediate field is annotated and completed to obtain a target field.

[0011] In some embodiments, the step of constructing an annotation dictionary based on the reference fields and the reference annotations includes:

[0012] Obtaining a mapping relationship between the reference field and the reference annotation;

[0013] Combining the reference field and the reference annotation according to the mapping relationship to obtain a key-value pair;

[0014] The annotation dictionary is obtained according to the key-value pair.

[0015] In some embodiments, the loading result includes a loading success or a loading failure, and the step of loading the annotation dictionary using the original field as a database key to obtain the loading result includes:

[0016] Calculating similarity between the database key and the key-value pair of the annotation dictionary to obtain field similarity;

[0017] If the field similarity is greater than or equal to a preset similarity threshold, the loading result is successful;

[0018] If the field similarity is less than a preset similarity threshold, the loading result is loading failure.

[0019] In some embodiments, the step of annotating the original field according to the reference annotation to obtain the target field includes:

[0020] Performing language conversion processing on the reference annotation to obtain a first annotation sentence;

[0021] The original field is annotated according to the first annotation statement to obtain the target field.

[0022] In some embodiments, the step of performing language conversion processing on the reference annotation to obtain the first annotation statement includes:

[0023] splicing the reference annotations according to a preset splicing order to obtain a first annotation text;

[0024] The first annotation text is subjected to language conversion processing to obtain the first annotation statement, wherein the first annotation statement is in a data definition language.

[0025] In some embodiments, if the loading result is a loading failure, performing format conversion processing on the original field to obtain an intermediate field, and annotating and completing the intermediate field to obtain the target field includes:

[0026] If the loading result is a loading failure, the original field is format-converted according to a preset field format to obtain the intermediate field;

[0027] Performing semantic completion on the middle field according to a preset semantic logic condition to obtain a second annotation statement;

[0028] The original field is annotated and supplemented according to the second annotation statement to obtain the target field.

[0029] In some embodiments, the step of performing semantic completion on the middle field according to a preset semantic logic condition to obtain a second annotation statement includes:

[0030] Storing the intermediate fields in a preset configuration table;

[0031] Performing semantic completion on the middle field using the configuration table and the semantic logic condition to obtain a second annotation text;

[0032] The second annotation text is subjected to language conversion processing to obtain the second annotation statement, wherein the second annotation statement is in a data definition language.

[0033] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides an annotation device, comprising:

[0034] A first reading module is used to read reference fields and reference annotations from a preset data warehouse;

[0035] A dictionary construction module is configured to construct an annotation dictionary based on the reference field and the reference annotation; wherein the annotation dictionary includes at least one key-value pair, and the key-value pair includes the reference field and the reference annotation in a one-to-one correspondence;

[0036] The second reading module is used to read the original field from the local library; wherein the annotation content of the original field is empty;

[0037] A loading module, configured to load the annotation dictionary using the original field as a database key to obtain a loading result;

[0038] an annotation processing module, configured to extract the reference annotation if the loading result is successful, and perform annotation processing on the original field according to the reference annotation to obtain a target field; wherein the target field includes field content and annotation content, the field content is derived from the original field, and the annotation content is derived from the reference annotation;

[0039] The annotation completion module is used to perform format conversion processing on the original field to obtain an intermediate field if the loading result is a loading failure, and perform annotation completion on the intermediate field to obtain a target field.

[0040] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, a program stored on the memory and runnable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the method described in the first aspect above is implemented.

[0041] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the first aspect above.

[0042] The present application proposes an annotation method, an annotation device, an electronic device, and a storage medium. The method reads reference fields and reference annotations from a preset data warehouse and constructs an annotation dictionary based on the reference fields and reference annotations. The annotation dictionary includes at least one key-value pair, and the key-value pair includes a one-to-one correspondence between a reference field and a reference annotation. This method can more conveniently construct an annotation dictionary for annotation completion, so that the field can be annotated according to the annotation dictionary to improve the accuracy of the annotation. Furthermore, the original field is read from a local library. The annotation content of the original field is empty. The annotation dictionary is loaded using the original field as a database key to obtain a loading result. In this way, it is convenient to find whether there is a reference annotation in the annotation dictionary that can match the original field. Specifically, if the loading result is successful, the reference annotation is extracted, and the original field is annotated based on the reference annotation to obtain a target field. The target field includes field content and annotation content, the field content is derived from the original field, and the annotation content is derived from the reference annotation. If the loading result is a loading failure, the original field is formatted to obtain an intermediate field, and the annotation of the intermediate field is completed to obtain the target field. In this way, the original fields can be annotated according to the annotation dictionary. At the same time, the original fields that do not have reference annotations in the annotation dictionary can also be annotated to improve the comprehensiveness and efficiency of annotations. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flowchart of the annotation method provided in an embodiment of the present application;

[0044] Figure 2 yes Figure 1 Flowchart of step S102 in FIG.

[0045] Figure 3 yes Figure 1 Flowchart of step S104 in FIG.

[0046] Figure 4 yes Figure 1 Flowchart of step S105 in FIG.

[0047] Figure 5 yes Figure 4 Flowchart of step S401 in FIG.

[0048] Figure 6 yes Figure 1 Flowchart of step S106 in FIG.

[0049] Figure 7 yes Figure 6 Flowchart of step S602 in FIG.

[0050] Figure 8 is a structural diagram of an annotation device provided in an embodiment of the present application;

[0051] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0053] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0054] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by those skilled in the art to which this application belongs. The terms used in this application are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0055] First, let’s analyze some of the terms used in this application:

[0056] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0057] Natural language processing (NLP): NLP uses computers to process, understand, and apply human languages ​​(such as Chinese and English). A branch of artificial intelligence, NLP is an interdisciplinary field between computer science and linguistics, often referred to as computational linguistics. Natural language processing encompasses grammatical analysis, semantic analysis, and discourse comprehension. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. It encompasses data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research related to language processing, and linguistics research related to language computing.

[0058] Information Extraction: A text processing technology that extracts specified types of entity, relationship, event, and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters. Text information is composed of some small specific units, such as characters, words, phrases, sentences, paragraphs, or a combination of these specific units. Extracting noun phrases, names, place names, etc. from text data is all text information extraction. Of course, the information extracted by text information extraction technology can be of various types.

[0059] MySQL Database: MySQL is a relational database management system. Relational databases store data in separate tables rather than storing all data in a single, large repository. This increases data processing speed and flexibility. The SQL language used by MySQL is the most commonly used, standardized language for accessing databases.

[0060] Index (database term): It is a data structure in the MySQL database, that is, a way of organizing data. This data structure is also called a key (primary key). In a relational database, an index is a separate, physical storage structure that sorts the values ​​of one or more columns in a database table. It is a collection of values ​​in one or more columns in a table and a corresponding list of logical pointers to the data pages in the table that physically identify these values. The role of an index is equivalent to the table of contents of a book, allowing you to quickly find the required content based on the page numbers in the table of contents. The index provides pointers to the data values ​​stored in the specified columns of the table, and then sorts these pointers according to the sort order you specify. The database uses the index to find a specific value and then follows the pointer to find the row containing that value. This allows SQL statements corresponding to the table to execute faster and quickly access specific information in the database table.

[0061] Primary Key: Also known as the primary keyword, it is one or more fields in a table whose values ​​are used to uniquely identify a record in the table. In a relationship between two tables, the primary key is used in one table to reference a specific record from another table. The primary key is a unique keyword that is part of the table definition. The primary key of a table can be composed of multiple keywords, and the primary keyword column cannot contain null values. The primary key is a column or a combination of columns whose values ​​can uniquely identify each row in the table, which can be used to enforce the entity integrity of the table. The primary key is mainly used to associate with foreign keys in other tables, as well as to modify and delete text records.

[0062] Data Definition Language (DDL) is a language used to describe real-world entities to be stored in a database.

[0063] Web crawler: Also known as a web spider or web robot, it is a program or script that automatically crawls the World Wide Web according to certain rules. Other less commonly used names include ant, automatic indexer, simulator, or worm.

[0064] Currently, manual annotation is often used when annotating fields in data tables. This method often requires manual scripting to annotate the original fields that lack annotations. This process involves manually writing scripts to complete the annotations, manually translating fields, and manually writing DDL statements. The workload is large and affects annotation efficiency. Therefore, how to improve annotation efficiency has become a technical problem that needs to be solved urgently.

[0065] Based on this, embodiments of the present application provide an annotation method, an annotation device, an electronic device, and a storage medium, aiming to improve annotation efficiency.

[0066] The annotation method, annotation device, electronic device and storage medium provided in the embodiments of the present application are specifically explained through the following embodiments. First, the annotation method in the embodiments of the present application is described.

[0067] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0068] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0069] The annotation method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The annotation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the annotation method, etc., but is not limited to the above forms.

[0070] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0071] Figure 1 This is an optional flowchart of the annotation method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0072] Step S101, reading reference fields and reference annotations from a preset data warehouse;

[0073] Step S102: construct an annotation dictionary based on the reference fields and the reference annotations; wherein the annotation dictionary includes at least one key-value pair, and the key-value pair includes a one-to-one correspondence between the reference fields and the reference annotations;

[0074] Step S103, reading the original field from the local library; wherein the annotation content of the original field is empty;

[0075] Step S104, loading the annotation dictionary using the original field as a database key to obtain a loading result;

[0076] Step S105: If the loading result is successful, the reference annotation is extracted, and the original field is annotated according to the reference annotation to obtain the target field; wherein the target field includes field content and annotation content, the field content is derived from the original field, and the annotation content is derived from the reference annotation;

[0077] Step S106: If the loading result is a loading failure, the original field is format converted to obtain an intermediate field, and the intermediate field is annotated and completed to obtain the target field.

[0078] In the embodiment of the present application, steps S101 to S106 are described, by reading reference fields and reference annotations from a preset data warehouse, and constructing an annotation dictionary based on the reference fields and reference annotations; wherein the annotation dictionary includes at least one key-value pair, and the key-value pair includes a one-to-one correspondence between a reference field and a reference annotation, which can more conveniently construct an annotation dictionary for annotation completion, so as to annotate the field according to the annotation dictionary and improve the accuracy of the annotation. Further, the original field is read from the local library; wherein the annotation content of the original field is empty; the annotation dictionary is loaded using the original field as the database key to obtain a loading result. In this way, it is convenient to find whether there is a reference annotation that can match the original field in the annotation dictionary. Specifically, if the loading result is a successful load, the reference annotation is extracted, and the original field is annotated based on the reference annotation to obtain a target field; wherein the target field includes field content and annotation content, the field content is derived from the original field, and the annotation content is derived from the reference annotation; if the loading result is a failed load, the original field is format converted to obtain an intermediate field, and the annotation of the intermediate field is completed to obtain the target field. In this way, the original fields can be annotated according to the annotation dictionary. At the same time, the original fields that do not have reference annotations in the annotation dictionary can also be annotated to improve the comprehensiveness and efficiency of annotations.

[0079] In step S101 of some embodiments, a script may be written to load a preset data warehouse, thereby reading reference fields and reference annotations from the data warehouse. The preset data warehouse may be a Hive database, which is a commonly used data warehouse tool in MySQL. The preset data warehouse may also be another database without limitation. The reference fields and reference annotations are generally strings, and the reference field is an annotated field. The reference annotations may be a random combination of Chinese, English, numbers, etc., without limitation.

[0080] See also Figure 2 In some embodiments, step S102 may include but is not limited to steps S201 to S203:

[0081] Step S201, obtaining a mapping relationship between reference fields and reference annotations;

[0082] Step S202: combining the reference fields and the reference annotations according to the mapping relationship to obtain a key-value pair;

[0083] Step S203: Obtain an annotation dictionary based on the key-value pairs.

[0084] In step S201 of some embodiments, field index information of the reference field and annotation index information of the reference annotation are obtained, wherein the field index information includes a field primary key that can represent the location of the reference field, and the annotation index information includes an annotation primary key that can represent the location of the reference annotation. Therefore, the row characteristics and column characteristics of the annotation primary key and the field primary key are compared. If, for a certain reference field and a certain reference annotation, the row characteristics of the annotation primary key are the same as the row characteristics of the field primary key, and the column characteristics of the annotation primary key are the same as the column characteristics of the field primary key, then it indicates that the reference field and the reference annotation are in the same position, and there is a mapping relationship between the two, that is, the reference annotation is the annotation of the reference field.

[0085] In step S202 of some embodiments, based on the mapping relationship between the reference field and the reference annotation, the reference field and the reference annotation at the same position are combined and associated, the reference field is used as the key data (i.e., key), and the reference annotation is used as the value data (i.e., value), and each pair of key data and value data with a mapping relationship is paired to form a key-value pair. In this way, the reference field and its corresponding reference annotation are represented in the form of a key-value pair, and multiple key-value pairs are obtained, wherein each reference field and each corresponding reference annotation constitute a key-value pair, and the reference field and reference annotation of each key-value pair correspond one to one.

[0086] In step S203 of some embodiments, the multiple key-value pairs obtained are stored as a whole in the database in the form of a data graph map, thereby obtaining an annotation dictionary.

[0087] In step S103 of some embodiments, a web crawler is pre-coded so that all tables of the local library are traversed by the web crawler, and the original fields are automatically captured from the local library; wherein the annotation content of the original field is empty, and the original field is generally represented in the form of a string; wherein the local library can be an Sx-hx-safe library or other databases, but is not limited thereto.

[0088] See also Figure 3 In some embodiments, the loading result includes loading success or loading failure. Step S104 may include but is not limited to steps S301 to S303:

[0089] Step S301, performing similarity calculation on the key-value pairs of the database key and the annotation dictionary to obtain field similarity;

[0090] Step S302: If the field similarity is greater than or equal to the preset similarity threshold, the loading result is successful.

[0091] Step S303: If the field similarity is less than a preset similarity threshold, the loading result is loading failure.

[0092] In step S301 of some embodiments, the key data (i.e., the reference field) in the key-value pair is extracted, and the database key (i.e., the original field) and the key data are similarity calculated by a preset similarity algorithm to obtain field similarity. Specifically, the database key is first mapped to a preset vector space to obtain the original key vector u, and the key data in the key-value pair is mapped to a preset vector space to obtain a reference key vector v, wherein the feature dimension of the preset vector space can be set according to actual business needs without limitation. Furthermore, the preset similarity algorithm may include a cosine similarity algorithm, etc. For example, the reference key vector and the original key vector are similarity calculated by a cosine similarity algorithm to obtain field similarity. The calculation process can be expressed as shown in formula (1):

[0093]

[0094] In step S302 of some embodiments, the preset similarity threshold can be set according to actual business needs without limitation. For example, the preset similarity threshold is 0.7. If the field similarity is greater than or equal to the similarity threshold, it indicates that the semantic information of the database key and the current key data are relatively close, and the semantics of the original field and the reference field of the key-value pair are similar. Therefore, the value data (reference annotation) corresponding to the reference field can be used as the annotation of the original field, that is, there is a reference annotation matching the original field in the annotation dictionary, and the original field can be annotated by the reference annotation. Therefore, the loading result is a successful load.

[0095] In step S303 of some embodiments, the preset similarity threshold can be set according to actual business needs without restriction. For example, the preset similarity threshold is 0.7. If the field similarity is less than the similarity threshold, it indicates that the semantic information of the database key and the current key data is quite different, and the semantic correlation between the original field and the reference field of the key-value pair is low. Therefore, it is not suitable to use the value data (reference annotation) corresponding to the reference field as the annotation of the original field, that is, there is no reference annotation matching the original field in the annotation dictionary, and other methods need to be used to annotate the original field. Therefore, the loading result is a loading failure.

[0096] Through the above steps S301 to S303, it is possible to more conveniently determine whether there is a reference annotation matching the original field in the annotation dictionary. When there is a reference annotation matching the original field in the annotation dictionary, the original field is annotated based on the annotation dictionary. When there is no reference annotation matching the original field in the annotation dictionary, other methods are used to complete the annotation, which can effectively improve the annotation efficiency and comprehensiveness.

[0097] See also Figure 4 In some embodiments, step S105 may include but is not limited to steps S401 to S402:

[0098] Step S401: performing language conversion processing on the reference annotation to obtain a first annotation statement;

[0099] Step S402: Annotate the original field according to the first annotation statement to obtain a target field.

[0100] In step S401 of some embodiments, if a reference annotation matching the original field exists in the annotation dictionary, indicating that the database key and the semantic information of the reference field are relatively close, the loading result is successful. Simultaneously, the annotation dictionary is traversed to extract the key-value pairs corresponding to the reference field, and the reference annotation is extracted from the key-value pairs. To improve annotation efficiency, the reference annotation in string form needs to be converted from a string to the database language to obtain a first annotation statement.

[0101] In step S402 of some embodiments, a field definition script is first preset, and attributes are added to the original field through the preset field definition script, wherein the added attributes are generally commet attributes, and then the first comment statement is added to the original field through the added commet attributes to implement comment processing of the original field and obtain the target field, wherein the target field includes field content and comment content, the field content comes from the original field, and the comment content comes from the reference comment.

[0102] Furthermore, in order to improve the rationality of the annotation, it is also necessary to obtain the annotation position corresponding to the original field. The annotation position is used to write the annotation content corresponding to the original field. The annotation position can be obtained by parsing the index information corresponding to the original field. Specifically, the TF-IDF algorithm can be used to extract annotation features from the index information. Each index information is processed into several character nodes. The frequency of each character in the index information is calculated using the TF-IDF algorithm to obtain the character's term frequency (TF), where TF = the number of times character W appears / the number of characters in the index information; further, the inverse document frequency (IDF) of each character is calculated, where IDF = log (total number of index information / (number of index information containing character w + 1)). Finally, the comprehensive frequency value of each character is calculated based on the term frequency and inverse document frequency. The comprehensive frequency value = term frequency * inverse document frequency, where character w is a type of word that can represent the meaning of annotation, mark, or mark. The node with the largest comprehensive frequency value and containing character w in the index information is selected as the annotation feature. The annotation feature can reflect the annotation position corresponding to the original field. Thus, the first comment statement is added to the annotation position of the original field through the commet attribute to obtain the target field, so that the target field can maintain a consistent format and improve the standardization of the target field.

[0103] See also Figure 5 In some embodiments, step S401 may include but is not limited to steps S501 to S502:

[0104] Step S501, splicing the reference annotations according to a preset splicing order to obtain a first annotation text;

[0105] Step S502 : performing language conversion processing on the first annotation text to obtain a first annotation statement, wherein the first annotation statement is in a data definition language.

[0106] In step S501 of some embodiments, the preset splicing order can be set according to actual business needs without limitation. For example, the preset splicing order can be to splice the reference annotations in the order of the time in which the reference annotations are obtained, or to arrange the original fields according to basic grammatical rules (such as commonly used subject-verb-object forms, etc.) to obtain the serial number of each original field, and to sort the corresponding reference annotations with the same serial number according to the size of the serial number of the original field, and to splice the sorted reference annotations to obtain the first annotation text.

[0107] In step S502 of some embodiments, in order to improve the rationality of the annotation and ensure that the annotation format meets business requirements, the first annotation text needs to be converted from a text string into a database language to obtain a first annotation statement. The database language is a DDL statement in the SQL language.

[0108] It should be noted that when performing language conversion on the first annotation text, it can be implemented by referring to the processing process of conventional Text-to-SQL tasks. For example, the potential knowledge of the first annotation text is learned through a deep learning model, and the relationship between the text sentences in the first annotation text is predicted, thereby generating a first annotation statement, wherein the first annotation statement is a DDL statement. The DDL statement is a data definition language commonly used in the MYSQL database. Using the form of DDL statements to annotate the original field can conveniently realize the storage of the annotation content, thereby improving the annotation efficiency and accuracy.

[0109] See also Figure 6 In some embodiments, step S106 includes but is not limited to steps S601 to S603:

[0110] Step S601: If the loading result is a loading failure, the original field is formatted according to a preset field format to obtain an intermediate field;

[0111] Step S602: semantically complete the middle field according to a preset semantic logic condition to obtain a second annotation statement;

[0112] Step S603: supplement the original field with annotations according to the second annotation statement to obtain the target field.

[0113] In step S601 of some embodiments, if there is no reference annotation matching the original field in the annotation dictionary, it indicates that the semantic information of the database key and a reference field is quite different, and it is not appropriate to use the reference annotation in the annotation dictionary as the annotation of the original field. Then the loading result is failure, and other methods need to be used to annotate the original field. In order to improve the processing efficiency of the original field that does not have a corresponding reference annotation match in the annotation dictionary, this type of original field can be marked. Specifically, the field name of the original field is formatted according to a preset field format to obtain an intermediate field, wherein the field name of the intermediate field can be expressed in the form of "field name of the original field + random number + autocommet". This can make the format of the intermediate field different from that of the original field, so that the original field that does not have a corresponding reference annotation match in the annotation dictionary can be filtered out from the local library.

[0114] In step S602 of some embodiments, a configuration table can be constructed based on all fields in the local library and their corresponding annotation contents. By traversing the configuration table, it is queried whether there is a field that matches the semantic characteristics of the middle field. If there is a field with matching semantic characteristics, the annotation content corresponding to the field in the configuration table is extracted according to the preset semantic logical conditions, and the middle field is semantically completed according to the annotation content to obtain a second annotation statement. Since the annotation content in the extracted configuration table may be discrete and not continuous annotation content that conforms to the semantic specification, it is necessary to splice these annotation contents in the form of subject, predicate, and object according to the semantic logical conditions to form a complete annotation statement, i.e., the second annotation statement.

[0115] In step S603 of some embodiments, a field definition script is first preset, and attributes are added to the original field through the preset field definition script, wherein the added attributes are generally commet attributes, and then the second comment statement is added to the original field through the added commet attributes to implement comment processing of the original field and obtain the target field.

[0116] See also Figure 7 In some embodiments, step S602 may include but is not limited to steps S701 to S703:

[0117] Step S701, storing the middle field in a preset configuration table;

[0118] Step S702: semantically complete the middle field using the configuration table and semantic logic conditions to obtain a second annotation text;

[0119] Step S703: performing language conversion processing on the second annotation text to obtain a second annotation statement, wherein the second annotation statement is in a data definition language.

[0120] In step S701 of some embodiments, the intermediate field is stored in a preset configuration table, wherein the configuration table includes multiple comment fields, and each comment field includes the table name of a data table in the local library, a field in the data table, and the comment content corresponding to the field; that is, the configuration table includes all fields of the local library and their corresponding comment content. It can be understood that the configuration table can be regarded as a dictionary constructed based on the fields of the local library and the comment content corresponding to the fields.

[0121] In step S702 of some embodiments, the configuration table includes a configuration switch, which is generally a dynamic switch. Through this configuration switch, the configuration table can be enabled or disabled according to actual business needs. For original fields that do not have corresponding reference annotation matches in the annotation dictionary, after the original field is formatted and the intermediate field is obtained, the configuration switch needs to be loaded into the startup state so that the configuration table can be enabled. The comment field in the configuration table is traversed and matched according to the intermediate field, and a comment field with a high similarity to the intermediate field is found, thereby extracting the annotation content in the comment field and using the annotation content as the annotation of the intermediate field. In this way, the intermediate field can be annotated more conveniently, and mutual complementation between data tables in the local library can be achieved, that is, the fields of another data table are annotated and supplemented according to the fields of a certain data table and the annotation content of the fields, thereby improving the comprehensiveness of the annotation.

[0122] For example, when field a of data table A in the local library does not have a corresponding reference annotation in the annotation dictionary, the format of field a is converted to obtain the intermediate field a1, and the configuration switch of the configuration table is loaded to the start state. The configuration table is searched for comment field 1 that is similar to the intermediate field a1. The content of comment field 1 is the table name of data table B in the local library, field b in data table B, and the annotation content corresponding to field b. This indicates that field a in data table A and field b in data table B may be fields representing the same semantic information. Therefore, the annotation content corresponding to field b can be used as the annotation content of field a.

[0123] Furthermore, since the annotation content in the extracted configuration table may be discrete and not continuous annotation content that conforms to semantic specifications, it is necessary to splice these annotation contents according to semantic logical conditions (for example, in the form of subject, predicate and object) to obtain a second annotation text.

[0124] In step S703 of some embodiments, to improve the rationality of the annotation and ensure that the annotation format meets business requirements, the second annotation text needs to be converted from a text string into a database language to obtain a second annotation statement. The database language is a DDL statement in the SQL language. DDL statements are a commonly used data definition language in MYSQL databases. Using DDL statements to annotate the original field can facilitate the storage of annotation content, thereby improving annotation efficiency and accuracy.

[0125] In addition, the annotation method of the embodiment of the present application can also perform annotation correction on the original field with annotation content in the local library to obtain the target field. Specifically, the annotation content of the original field is first checked for eligibility to check whether the annotation content of the original field is garbled or space, etc. If the annotation content is a normal string, it is assumed that the annotation content of the original field is correct. If the annotation content of the original field is garbled or space, etc., the original field is re-annotated according to the annotation process of the above steps S104, S105, and S106 to obtain the target field.

[0126] The annotation method of the embodiment of the present application reads reference fields and reference annotations from a preset data warehouse, and constructs an annotation dictionary based on the reference fields and reference annotations; wherein the annotation dictionary includes at least one key-value pair, and the key-value pair includes a one-to-one corresponding reference field and reference annotation, which can more conveniently construct an annotation dictionary for annotation completion, so as to annotate the field according to the annotation dictionary and improve the accuracy of the annotation. Further, the original field is read from the local library; wherein the annotation content of the original field is empty; the annotation dictionary is loaded with the original field as the database key to obtain a loading result. In this way, it is convenient to find whether there is a reference annotation that can match the original field in the annotation dictionary. Specifically, if the loading result is a successful load, the reference annotation is extracted, and the original field is annotated according to the reference annotation to obtain a target field; wherein the target field includes field content and annotation content, the field content comes from the original field, and the annotation content comes from the reference annotation; if the loading result is a failed load, the original field is format converted to obtain an intermediate field, and the intermediate field is annotated to obtain the target field. In this way, the original fields can be annotated according to the annotation dictionary. At the same time, the original fields that do not have reference annotations in the annotation dictionary can also be annotated to improve the comprehensiveness and efficiency of annotations.

[0127] See also Figure 8 , the embodiment of the present application further provides an annotation device that can implement the above annotation method, the device comprising:

[0128] The first reading module 801 is used to read reference fields and reference annotations from a preset data warehouse;

[0129] A dictionary construction module 802 is configured to construct an annotation dictionary based on the reference fields and the reference annotations; wherein the annotation dictionary includes at least one key-value pair, and the key-value pair includes a one-to-one correspondence between the reference fields and the reference annotations;

[0130] The second reading module 803 is used to read the original field from the local library; wherein the annotation content of the original field is empty;

[0131] The loading module 804 is used to load the annotation dictionary using the original field as a database key to obtain a loading result;

[0132] The annotation processing module 805 is used to extract the reference annotation if the loading result is successful, and perform annotation processing on the original field based on the reference annotation to obtain the target field; wherein the target field includes field content and annotation content, the field content is derived from the original field, and the annotation content is derived from the reference annotation;

[0133] The annotation completion module 806 is used to perform format conversion processing on the original field to obtain an intermediate field if the loading result is a loading failure, and perform annotation completion on the intermediate field to obtain a target field.

[0134] In some embodiments, the dictionary building module 802 includes:

[0135] A mapping unit, used to obtain the mapping relationship between reference fields and reference annotations;

[0136] A combination unit is used to combine the reference fields and reference annotations according to the mapping relationship to obtain a key-value pair;

[0137] The dictionary generation unit is used to obtain an annotation dictionary based on key-value pairs.

[0138] In some embodiments, the loading result includes loading success or loading failure, and the loading module 804 includes:

[0139] A similarity calculation unit is used to calculate the similarity between the database key and the key-value pair of the annotation dictionary to obtain the field similarity;

[0140] The first loading unit is configured to determine that the loading result is successful if the field similarity is greater than or equal to a preset similarity threshold;

[0141] The second loading unit is configured to determine that the loading result is a loading failure if the field similarity is less than a preset similarity threshold.

[0142] In some embodiments, the annotation processing module 805 includes:

[0143] A language conversion unit, configured to perform language conversion processing on the reference annotation to obtain a first annotation statement;

[0144] The annotation unit is used to perform annotation processing on the original field according to the first annotation statement to obtain the target field.

[0145] In some embodiments, the language conversion unit includes:

[0146] A splicing subunit, configured to splice the reference annotations according to a preset splicing order to obtain a first annotation text;

[0147] The first language conversion sub-unit is configured to perform language conversion processing on the first annotation text to obtain a first annotation statement, wherein the first annotation statement is a DDL statement.

[0148] In some embodiments, the annotation completion module 806 includes:

[0149] A format conversion unit, configured to convert the format of the original field according to a preset field format to obtain an intermediate field if the loading result is a loading failure;

[0150] A semantic completion unit, configured to perform semantic completion on the middle field according to a preset semantic logic condition to obtain a second annotation statement;

[0151] The annotation supplement unit is used to supplement the original field with annotations according to the second annotation statement to obtain a target field.

[0152] In some embodiments, the semantic completion unit includes:

[0153] A storage subunit, used for storing intermediate fields into a preset configuration table;

[0154] The completion subunit is used to perform semantic completion on the middle field through the configuration table and semantic logic conditions to obtain the second annotation text;

[0155] The second language conversion sub-unit is used to perform language conversion processing on the second annotation text to obtain a second annotation statement, wherein the second annotation statement is a DDL statement.

[0156] The specific implementation of the annotation device is basically the same as the specific embodiment of the above-mentioned annotation method, and will not be repeated here.

[0157] The present application also provides an electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the program is executed by the processor, the aforementioned annotation method is implemented. The electronic device may be any intelligent terminal, such as a tablet computer or an in-vehicle computer.

[0158] See also Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0159] The processor 901 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0160] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program codes are stored in the memory 902 and are called by the processor 901 to execute the annotation method of the embodiments of this application.

[0161] Input / output interface 903, used to implement information input and output;

[0162] Communication interface 904, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0163] Bus 905 , which transmits information between various components of the device (e.g., processor 901 , memory 902 , input / output interface 903 , and communication interface 904 );

[0164] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0165] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned annotation method.

[0166] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0167] The annotation method, annotation device, electronic device and storage medium provided by the embodiment of the present application read reference fields and reference annotations from a preset data warehouse, and construct an annotation dictionary based on the reference fields and reference annotations; wherein the annotation dictionary includes at least one key-value pair, and the key-value pair includes a one-to-one corresponding reference field and reference annotation, which can more conveniently construct an annotation dictionary for annotation completion, so as to annotate the field according to the annotation dictionary and improve the accuracy of the annotation. Further, the original field is read from the local library; wherein the annotation content of the original field is empty; the annotation dictionary is loaded with the original field as the database key to obtain a loading result. In this way, it is convenient to find whether there is a reference annotation that can match the original field in the annotation dictionary. Specifically, if the loading result is a successful load, the reference annotation is extracted, and the original field is annotated according to the reference annotation to obtain a target field; wherein the target field includes field content and annotation content, the field content comes from the original field, and the annotation content comes from the reference annotation; if the loading result is a failed load, the original field is format converted to obtain an intermediate field, and the intermediate field is annotated to obtain the target field. In this way, the original fields can be annotated according to the annotation dictionary. At the same time, the original fields that do not have reference annotations in the annotation dictionary can also be annotated to improve the comprehensiveness and efficiency of annotations.

[0168] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0169] It will be understood by those skilled in the art that Figure 1-7 The technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figures, or a combination of certain steps, or different steps.

[0170] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0171] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0172] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0173] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0174] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0175] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0176] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0177] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0178] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. An annotation method, characterized in that: The method comprises: Read reference fields and reference annotations from a preset data warehouse; construct an annotation dictionary based on the reference fields and the reference annotations; wherein the annotation dictionary includes at least one key-value pair, and the key-value pair includes the reference fields and the reference annotations in a one-to-one correspondence; Reading an original field from a local library; wherein the annotation content of the original field is empty; Loading the annotation dictionary using the original field as a database key to obtain a loading result; If the loading result is successful, the reference annotation is extracted, and the original field is annotated according to the reference annotation to obtain a target field; wherein the target field includes field content and annotation content, the field content is derived from the original field, and the annotation content is derived from the reference annotation; If the loading result is a loading failure, the original field is format converted to obtain an intermediate field, and the intermediate field is annotated and completed to obtain a target field; If the loading result is a loading failure, the original field is format converted to obtain an intermediate field, and the intermediate field is annotated and completed to obtain a target field, including: If the loading result is a loading failure, the field name of the original field is formatted according to a preset field format to obtain the intermediate field, where the field name of the intermediate field is represented as a combination of the field name of the original field, a random number, and an autocomment; The intermediate field is stored in a preset configuration table, wherein the configuration table includes multiple comment fields, each comment field includes a table name, a field, and a comment content corresponding to a data table in a local library; The configuration table is traversed to check whether there is a comment field that matches the semantic characteristics of the middle field. If there is a comment field that matches the semantic characteristics, the comment content corresponding to the comment field in the configuration table is extracted according to a preset semantic logic condition, the middle field is semantically completed according to the comment content to obtain a second comment text, and the second comment text is converted from a text string into a database language to obtain a second comment statement, wherein the database language is a DDL statement in the SQL language; A comment attribute is added to the original field through a preset field definition script, and a second comment statement is added to the original field through the added comment attribute to obtain the target field.

2. The annotation method according to claim 1, characterized in that The step of constructing an annotation dictionary according to the reference fields and the reference annotations comprises: Obtaining a mapping relationship between the reference field and the reference annotation; Combining the reference field and the reference annotation according to the mapping relationship to obtain a key-value pair; The annotation dictionary is obtained according to the key-value pair.

3. The annotation method according to claim 1, characterized in that The step of loading the annotation dictionary using the original field as a database key to obtain a loading result includes: Calculating similarity between the database key and the key-value pair of the annotation dictionary to obtain field similarity; If the field similarity is greater than or equal to a preset similarity threshold, the loading result is successful; If the field similarity is less than a preset similarity threshold, the loading result is loading failure.

4. The annotation method according to claim 1, characterized in that The step of annotating the original field according to the reference annotation to obtain the target field includes: Performing language conversion processing on the reference annotation to obtain a first annotation sentence; The original field is annotated according to the first annotation statement to obtain the target field.

5. The annotation method according to claim 4, characterized in that The step of performing language conversion processing on the reference annotation to obtain a first annotation statement includes: splicing the reference annotations according to a preset splicing order to obtain a first annotation text; The first annotation text is subjected to language conversion processing to obtain the first annotation statement, wherein the first annotation statement is in a data definition language.

6. An annotation device, characterized in that: The device comprises: A first reading module is used to read reference fields and reference annotations from a preset data warehouse; A dictionary construction module is configured to construct an annotation dictionary based on the reference field and the reference annotation; wherein the annotation dictionary includes at least one key-value pair, and the key-value pair includes the reference field and the reference annotation in a one-to-one correspondence; The second reading module is used to read the original field from the local library; wherein the annotation content of the original field is empty; A loading module, configured to load the annotation dictionary using the original field as a database key to obtain a loading result; an annotation processing module, configured to extract the reference annotation if the loading result is successful, and perform annotation processing on the original field according to the reference annotation to obtain a target field; wherein the target field includes field content and annotation content, the field content is derived from the original field, and the annotation content is derived from the reference annotation; An annotation completion module is used for, if the loading result is a loading failure, performing format conversion processing on the original field to obtain an intermediate field, and performing annotation completion on the intermediate field to obtain a target field; If the loading result is a loading failure, the original field is format converted to obtain an intermediate field, and the intermediate field is annotated and completed to obtain a target field, including: If the loading result is a loading failure, the field name of the original field is formatted according to a preset field format to obtain the intermediate field, where the field name of the intermediate field is represented as a combination of the field name of the original field, a random number, and an autocomment; The intermediate field is stored in a preset configuration table, wherein the configuration table includes multiple comment fields, each comment field includes a table name, a field, and a comment content corresponding to a data table in a local library; The configuration table is traversed to check whether there is a comment field that matches the semantic characteristics of the middle field. If there is a comment field that matches the semantic characteristics, the comment content corresponding to the comment field in the configuration table is extracted according to a preset semantic logic condition, the middle field is semantically completed according to the comment content to obtain a second comment text, and the second comment text is converted from a text string into a database language to obtain a second comment statement, wherein the database language is a DDL statement in the SQL language; A comment attribute is added to the original field through a preset field definition script, and a second comment statement is added to the original field through the added comment attribute to obtain the target field.

7. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the annotation method according to any one of claims 1 to 5 are realized.

8. A storage medium, wherein the storage medium is a computer-readable storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any one of claims 1 to 5.

Images