A method and device for constructing a label database

Abstract

The present invention provides a method and device for constructing a label database, including: obtaining open source code information; parsing the open source code information through an abstract syntax tree to obtain feature statements; constructing a dependency module database according to the feature statements; extracting labels from dependency modules in the dependency module database to obtain candidate labels; selecting technical labels from the candidate labels; determining technical fields and implementation functions according to the technical labels; wherein the dependency modules are used to characterize statements for introducing third-party libraries, and the workload of manual collection can be reduced by constructing a dependency module database, labels are calibrated through the dependency module database to obtain candidate labels, and judgment or correction is performed according to the candidate labels to improve efficiency and accuracy.

Description

Technical Field

[0001] The present invention relates to the field of software technology, and in particular to a method and device for constructing a label database. Background Art

[0002] At present, various third-party modules may be referenced in software codes to implement software functions. These third-party modules involve different technical fields. Generally, the software's dependent modules are manually collected, and the software's dependent modules are labeled. The technical field to which the software's dependent modules belong is determined by labeling. Specifically, the dependent library list corresponding to each dependent module in the software is collected, and the technical field to which the dependent module belongs is determined based on the relevant information in the dependent library list.

[0003] The above collection process is done manually, which is labor-intensive and requires manual label calibration. Since each person has different understandings of the technical field, label calibration varies greatly, thereby reducing accuracy. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a method and device for constructing a label database. By constructing a dependent module database, the workload of manual collection can be reduced. The labels can be calibrated by the dependent module database to obtain alternative labels, and judgment or correction can be made based on the alternative labels to improve efficiency and accuracy.

[0005] In a first aspect, an embodiment of the present invention provides a method for constructing a tag database, the method comprising:

[0006] Get open source code information;

[0007] Parsing the open source code information through an abstract syntax tree to obtain a feature statement;

[0008] Building a dependency module database according to the feature statement;

[0009] Extracting labels from dependent modules in the dependent module database to obtain candidate labels;

[0010] Selecting a technical label from the candidate labels;

[0011] Determine the technical field and implementation function based on the technical label;

[0012] The dependency module is used to represent the statement of introducing the third-party library.

[0013] Furthermore, the step of constructing a dependency module database according to the feature statement includes:

[0014] Filtering the built-in modules in the feature statement to obtain a feature statement including the dependent module;

[0015] Constructing the dependency module database according to the feature statements including the dependency modules;

[0016] The built-in module is used to represent statements that are unrelated to the characteristics of the technical field.

[0017] Furthermore, extracting labels from the dependent modules in the dependent module database to obtain candidate labels includes:

[0018] Using the name of the dependent module as a keyword, obtaining a search content information set corresponding to the dependent module through search technology and crawler technology;

[0019] Segmenting the search content information set using a word segmentation technique to obtain a plurality of word segments;

[0020] Extracting the multiple word segments according to a preset technical field label set to obtain multiple labels;

[0021] The multiple labels are used as the candidate labels.

[0022] Furthermore, the multiple word segments are extracted according to a preset technical field label set to obtain multiple labels, including:

[0023] Sorting the multiple participles according to the number of repetitions from large to small to obtain a plurality of sorted participles;

[0024] The sorted multiple word segments are extracted through the preset technical field tag set to obtain the multiple tags.

[0025] In a second aspect, an embodiment of the present invention provides a device for constructing a tag database, the device comprising:

[0026] Acquisition module, used to obtain open source code information;

[0027] A parsing module, used to parse the open source code information through an abstract syntax tree to obtain a feature statement;

[0028] A construction module, used to construct a dependent module database according to the feature statement;

[0029] An extraction module, used to extract labels from dependent modules in the dependent module database to obtain candidate labels;

[0030] A selection module, used for selecting a technical label from the candidate labels;

[0031] A determination module, used to determine the technical field and implement the function according to the technical label;

[0032] The dependency module is used to represent the statement of introducing the third-party library.

[0033] Furthermore, the building blocks are specifically used for:

[0034] Filtering the built-in modules in the feature statement to obtain a feature statement including the dependent module;

[0035] Constructing the dependency module database according to the feature statements including the dependency modules;

[0036] The built-in module is used to represent statements that are unrelated to the characteristics of the technical field.

[0037] Furthermore, the extraction module is specifically used for:

[0038] Using the name of the dependent module as a keyword, obtaining a search content information set corresponding to the dependent module through search technology and crawler technology;

[0039] Segmenting the search content information set using a word segmentation technique to obtain a plurality of word segments;

[0040] Extracting the multiple word segments according to a preset technical field label set to obtain multiple labels;

[0041] The multiple labels are used as the candidate labels.

[0042] Furthermore, the extraction module is specifically used for:

[0043] Sorting the multiple participles according to the number of repetitions from large to small to obtain a plurality of sorted participles;

[0044] The sorted multiple word segments are extracted through the preset technical field tag set to obtain the multiple tags.

[0045] In a third aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the method described above when executing the computer program.

[0046] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium having a non-volatile program code executable by a processor, wherein the program code enables the processor to execute the method as described above.

[0047] The embodiment of the present invention provides a method and device for constructing a label database, including: obtaining open source code information; parsing the open source code information through an abstract syntax tree to obtain feature statements; constructing a dependency module database according to the feature statements; extracting labels from dependency modules in the dependency module database to obtain candidate labels; selecting technical labels from the candidate labels; determining the technical field and implementation functions according to the technical labels; wherein the dependency module is used to characterize statements for introducing third-party libraries, and the workload of manual collection can be reduced by constructing a dependency module database, and labels are calibrated through the dependency module database to obtain candidate labels, and judgment or correction is performed according to the candidate labels to improve efficiency and accuracy.

[0048] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0049] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A flowchart of a method for constructing a tag database provided in Embodiment 1 of the present invention;

[0052] Figure 2 This is a flowchart of step S103 in a method for building a tag database provided in the first embodiment of the present invention;

[0053] Figure 3 A flowchart of step S104 in a method for building a tag database provided in the first embodiment of the present invention;

[0054] Figure 4 A schematic diagram of a device for constructing a label database provided in Embodiment 2 of the present invention.

[0055] icon:

[0056] 1- Acquisition module; 2- Parsing module; 3- Construction module; 4- Extraction module; 5- Selection module; 6- Determination module. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] To facilitate understanding of this embodiment, the embodiment of the present invention is described in detail below.

[0059] Embodiment 1:

[0060] Figure 1 A flow chart of a method for constructing a tag database provided in Embodiment 1 of the present invention.

[0061] Reference Figure 1 , the method comprises the following steps:

[0062] Step S101, obtaining open source code information;

[0063] Step S102, parsing the open source code information through an abstract syntax tree to obtain a feature statement;

[0064] Specifically, the open source code information is parsed through an abstract syntax tree to obtain a feature statement, which is a structured tree object (including multiple parent and child nodes). For example, a third-party library is introduced through syntax such as import and include.

[0065] Among them, feature statements include dependent modules and built-in modules. Dependent modules are used to represent statements that introduce third-party libraries. Built-in modules are used to represent statements that are not related to technical field features. Therefore, when constructing a dependent module database, feature statements need to filter out built-in modules. For example, the import os statement in Python language, the os module is a built-in module and needs to be filtered out.

[0066] Step S103, constructing a dependent module database according to the feature statement;

[0067] Specifically, the dependency module database includes dependency modules and the heat of dependency modules. Heat refers to the number of times a dependency module is referenced in a feature statement. The more times it is referenced, the higher the heat. When performing label processing, dependency modules with high heat can be given priority. Building a dependency module database based on feature statements can reduce the workload of manual collection and manual entry through various methods, and the collected content is richer and more extensive.

[0068] You can also use crawler technology to collect dependent module databases from corresponding language module management systems such as Maven and PyPI, but you need to perform different processing based on the syntax and usage scenarios of the modules introduced in different languages ​​before you can build a dependent module database.

[0069] Step S104, extracting labels from dependent modules in the dependent module database to obtain candidate labels;

[0070] Here, each dependent module in the dependent module database is labeled to obtain candidate labels, and the technical field and implementation function of the dependent module can be determined according to the candidate labels.

[0071] Step S105, selecting a technical tag from the candidate tags;

[0072] Step S106, determining the technical field and implementation function according to the technical tag.

[0073] Specifically, technical labels close to the technical field are selected from the candidate labels, which can reduce the workload of manual labeling and improve the accuracy of label selection.

[0074] Further, refer to Figure 2 , step S103 includes the following steps:

[0075] Step S201, filtering the built-in modules in the characteristic sentence to obtain the characteristic sentence including the dependent module;

[0076] Step S202, constructing a dependent module database according to the characteristic statements including the dependent modules;

[0077] Among them, the built-in module is used to represent statements that are not related to the characteristics of the technical field.

[0078] Here, since the built-in modules in the feature sentences are sentences unrelated to the technical field features, it is necessary to filter the built-in modules to obtain feature sentences including dependent modules.

[0079] Further, see Figure 3 , step S104 includes the following steps:

[0080] Step S301, using the name of the dependent module as a keyword, obtaining a search content information set corresponding to the dependent module through search technology and crawler technology;

[0081] Step S302, segmenting the search content information set using a word segmentation technique to obtain a plurality of word segments;

[0082] Step S303, extracting multiple word segments according to a preset technical field label set to obtain multiple labels;

[0083] Step S304: taking multiple tags as candidate tags.

[0084] Specifically, the name of the dependent module is used as a keyword to retrieve relevant content through search engines (such as Magi and Baidu), and the search content information set corresponding to the dependent module is obtained in combination with crawler technology; crawler technology is a program or script that automatically captures World Wide Web information according to certain rules.

[0085] The search content information set is segmented by word segmentation technology to obtain multiple word segments. For example, "what field does new energy vehicle belong to" is segmented into "new energy", "vehicle", "belong to", "what", and "field".

[0086] By presetting a set of technical field labels to assist label extraction, the relevance and accuracy of labels are improved by increasing the weight of labels, and ultimately multiple labels are obtained and used as candidate labels.

[0087] Further, step S303 includes the following steps:

[0088] Step S401, sorting the multiple segmented words according to the number of repetitions from large to small to obtain the sorted multiple segmented words;

[0089] Step S402 , extracting the sorted multiple segmented words through a preset technical field tag set to obtain multiple tags.

[0090] Specifically, multiple participles are sorted from large to small according to the number of repetitions to obtain multiple sorted participles; the sorted multiple participles are extracted through a preset technical field label set to obtain multiple labels with high relevance to the technical field and a large number of repetitions, and the multiple labels are used as alternative labels.

[0091] The embodiment of the present invention provides a method for constructing a label database, including: obtaining open source code information; parsing the open source code information through an abstract syntax tree to obtain feature statements; constructing a dependency module database according to the feature statements; extracting labels from dependency modules in the dependency module database to obtain candidate labels; selecting technical labels from the candidate labels; determining the technical field and implementation functions according to the technical labels; wherein the dependency module is used to characterize statements for introducing third-party libraries, and the workload of manual collection can be reduced by constructing a dependency module database, and labels are calibrated through the dependency module database to obtain candidate labels, and judgment or correction is performed according to the candidate labels to improve efficiency and accuracy.

[0092] Embodiment 2:

[0093] Figure 4 A schematic diagram of a device for constructing a label database provided in Embodiment 2 of the present invention.

[0094] Reference Figure 4 , the device comprises:

[0095] Acquisition module 1, used to obtain open source code information;

[0096] Parsing module 2, used to parse the open source code information through an abstract syntax tree to obtain feature sentences;

[0097] Construction module 3, used to construct a dependency module database according to the feature statement;

[0098] Extraction module 4, used for extracting labels from dependent modules in the dependent module database to obtain candidate labels;

[0099] A selection module 5 is used to select a technical label from the candidate labels;

[0100] A determination module 6, used to determine the technical field and implement the function according to the technical label;

[0101] The dependency module is used to represent the statement of introducing a third-party library.

[0102] Furthermore, building block 3 is specifically used for:

[0103] Filter the built-in modules in the feature statements to obtain feature statements including dependent modules;

[0104] Building a dependency module database according to feature statements including dependency modules;

[0105] Among them, the built-in module is used to represent statements that are not related to the characteristics of the technical field.

[0106] Furthermore, the extraction module 4 is specifically used for:

[0107] Using the name of the dependent module as a keyword, the search content information set corresponding to the dependent module is obtained through search technology and crawler technology;

[0108] The search content information set is segmented by word segmentation technology to obtain multiple word segments;

[0109] Extract multiple word segments according to a preset technical field label set to obtain multiple labels;

[0110] A plurality of labels are used as the candidate labels.

[0111] Furthermore, the extraction module 4 is specifically used for:

[0112] Sort multiple participles according to the number of repetitions from large to small to obtain sorted participles;

[0113] The sorted multiple word segments are extracted through a preset technical field label set to obtain multiple labels.

[0114] An embodiment of the present invention provides a device for constructing a label database, including: obtaining open source code information; parsing the open source code information through an abstract syntax tree to obtain feature statements; constructing a dependency module database according to the feature statements; extracting labels from dependency modules in the dependency module database to obtain candidate labels; selecting technical labels from the candidate labels; determining technical fields and implementation functions according to the technical labels; wherein the dependency modules are used to characterize statements that introduce third-party libraries, and the workload of manual collection can be reduced by constructing a dependency module database, and labels are calibrated through the dependency module database to obtain candidate labels, and judgment or correction is performed according to the candidate labels to improve efficiency and accuracy.

[0115] An embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a method for constructing a label database provided in the above embodiment are implemented.

[0116] An embodiment of the present invention further provides a computer-readable medium having a non-volatile program code executable by a processor, wherein a computer program is stored on the computer-readable medium, and when the computer program is executed by the processor, the steps of a method for constructing a tag database in the above embodiment are executed.

[0117] The computer program product provided in the embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be found in the method embodiment and will not be repeated here.

[0118] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0119] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0120] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0121] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0122] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for constructing a label database, characterized in that: The method comprises: Get open source code information; Parsing the open source code information through an abstract syntax tree to obtain a feature statement; Building a dependency module database according to the feature statement; Extracting labels from dependent modules in the dependent module database to obtain candidate labels; Selecting a technical label from the candidate labels; Determine the technical field and implementation function based on the technical label; Wherein, the dependency module is used to represent the statement of introducing the third-party library; The characteristic statement is a structured tree object, and the dependent module database includes the dependent modules and the heat of the dependent modules; the heat is the number of times the dependent module is cited in the characteristic statement; The step of extracting labels from the dependent modules in the dependent module database to obtain candidate labels includes: Using the name of the dependent module as a keyword, obtaining a search content information set corresponding to the dependent module through search technology and crawler technology; Segmenting the search content information set using a word segmentation technique to obtain a plurality of word segments; Extracting the multiple word segments according to a preset technical field label set to obtain multiple labels; Using the multiple labels as the candidate labels; The step of constructing a dependency module database according to the feature statement includes: Filtering the built-in modules in the feature statement to obtain a feature statement including the dependent module; Constructing the dependency module database according to the feature statements including the dependency modules; The built-in module is used to represent statements that are unrelated to the characteristics of the technical field.

2. A method for constructing a tag database according to claim 1, characterized in that: The multiple word segments are extracted according to a preset technical field tag set to obtain multiple tags, including: Sorting the multiple participles according to the number of repetitions from large to small to obtain a plurality of sorted participles; The sorted multiple word segments are extracted through the preset technical field tag set to obtain the multiple tags.

3. A device for constructing a label database, characterized in that: The device comprises: Acquisition module, used to obtain open source code information; A parsing module, used to parse the open source code information through an abstract syntax tree to obtain a feature statement; A construction module, used to construct a dependent module database according to the feature statement; An extraction module, used to extract labels from dependent modules in the dependent module database to obtain candidate labels; A selection module, used for selecting a technical label from the candidate labels; A determination module, used to determine the technical field and implement the function according to the technical label; Wherein, the dependency module is used to represent the statement of introducing the third-party library; The characteristic statement is a structured tree object, and the dependent module database includes the dependent modules and the heat of the dependent modules; the heat is the number of times the dependent module is cited in the characteristic statement; The extraction module is specifically used for: Using the name of the dependent module as a keyword, obtaining a search content information set corresponding to the dependent module through search technology and crawler technology; Segmenting the search content information set using a word segmentation technique to obtain a plurality of word segments; Extracting the multiple word segments according to a preset technical field label set to obtain multiple labels; Using the multiple labels as the candidate labels; The building blocks are specifically used for: Filtering the built-in modules in the feature statement to obtain a feature statement including the dependent module; Constructing the dependency module database according to the feature statements including the dependency modules; The built-in module is used to represent statements that are unrelated to the characteristics of the technical field.

4. The device for constructing a label database according to claim 3, characterized in that: The extraction module is specifically used for: Sorting the multiple participles according to the number of repetitions from large to small to obtain a plurality of sorted participles; The sorted multiple word segments are extracted through the preset technical field tag set to obtain the multiple tags.

5. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the computer program, the method according to claim 1 or 2 is implemented.

6. A computer-readable medium having a non-volatile program code executable by a processor, characterized in that: The program code causes the processor to execute the method of claim 1 or 2.

Images