Data processing method and system for online task execution script

Abstract

The application provides a data processing method and system for online task execution scripts, comprising: collecting online script execution task records; constructing an original database according to the online script execution task records; parsing standard format data stored in the original database to obtain a parsing result, the parsing result comprising character names, field appearance frequencies, table name appearance frequencies, table-to-table relationships and field-to-field relationships; statistically processing the parsing result to obtain appearance frequencies and accompanying lists; obtaining character names input by a user; retrieving corresponding character names from the accompanying lists according to the character names and generating a candidate list; arranging the character names in the candidate list in descending order according to the appearance frequencies to obtain an arranged candidate list; and screening the appearance probabilities of table name characters and field name characters according to the parsing result, so as to reduce the time for the user to select in the prompt list and improve the user experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a data processing method and system of online task execution script. BACKGROUND

[0002] With the vigorous development of big data technology, in order to improve the development efficiency of big data script, various big data development IDE (Integrated Development Environment) has continuously introduced code prompt function.

[0003] When the table name prompt is performed, the following method is adopted: when the from keyword is input during the writing of the sql script, the IDE will retrieve all the table names in the current database of the system and cache them into a preselected table name list. According to the input table name character, the retrieval is performed in the preselected table name list, the matched table names are retrieved in order, and the retrieval list is displayed in alphabetical order.

[0004] When the field name prompt is performed, the following method is adopted: after the select keyword is input, the IDE will frame the range of the alternative field name according to the table name after the from keyword and the table alias; after the field character is input, the retrieval is performed in the range of the alternative field name, and the retrieval result is displayed in alphabetical order according to the first letter.

[0005] The existing script prompt method can improve the coding speed of the user, but still needs to be distinguished and selected after the prompt. In terms of table name matching, each table needs to be matched in order, and if there are a large number of tables and the table names are similar, the selection time of the user will be increased. For example, there are 1000 tables of user-a1, user-a2, user-a3…user-a1000 in the database, and after the user-a is input, 1000 tables are listed and displayed in order. The user needs to distinguish and select among the 1000 tables, and cannot accurately match or further narrow the selection range; if the association query between multiple tables is involved, each table needs to be retrieved separately, and the association relationship between the tables cannot be analyzed and accurate prompt cannot be given. SUMMARY

[0006] Therefore, the purpose of the present application is to provide a data processing method and system of online task execution script, which screens the appearance probability of the table name character and the field name according to the analysis result, reduces the selection time of the user in the prompt list, and improves the user experience.

[0007] In a first aspect, the embodiments of the present application provide a data processing method of online task execution script, which comprises:

[0008] Collecting an online script execution task record;

[0009] Building an original database according to the online script execution task record;

[0010] Parsing standard format data stored in the original database to obtain a parsing result, the parsing result including character name, field occurrence frequency, table occurrence frequency, inter-table relationship and inter-field relationship;

[0011] Counting the parsing result to obtain occurrence frequency and accompanying list;

[0012] Obtaining the character name input by a user;

[0013] Retrieving the corresponding character name from the accompanying list according to the character name and generating an alternative list;

[0014] Arranging the character names in the alternative list in descending order according to the occurrence frequency to obtain an arranged alternative list.

[0015] Further, the online script execution task record includes multiple execution records, and building an original database according to the online script execution task record includes:

[0016] Marking the multiple execution records respectively to obtain multiple marked execution records;

[0017] Organizing the multiple marked execution records to obtain original data in standard format;

[0018] Storing the original data in standard format into a database to generate the original database.

[0019] Further, the occurrence frequency includes table occurrence frequency and field occurrence frequency, and the accompanying list includes accompanying table list and accompanying field list; counting the parsing result to obtain occurrence frequency and accompanying list includes:

[0020] Counting the table occurrence frequency to obtain the table occurrence frequency;

[0021] Counting the field occurrence frequency to obtain the field occurrence frequency;

[0022] According to the inter-table relationship, counting the frequency of other tables corresponding to the current table when the current table appears, and generating a list of the other tables as the accompanying table list of the current table;

[0023] According to the inter-field relationship, counting the frequency of other fields corresponding to the current field when the current field appears, and generating a list of the other fields as the accompanying field list of the current field.

[0024] Furthermore, the character name includes table name characters and field names; based on the character name, the corresponding character name is retrieved from the accompanying list, and a candidate list is generated, including:

[0025] Based on the table name characters, retrieve the corresponding table name characters from the accompanying table list and generate a candidate table name list;

[0026] or,

[0027] Based on the field name, the corresponding field name is retrieved from the accompanying field list, and a candidate field list is generated.

[0028] Furthermore, the character names in the candidate list are arranged in descending order according to their frequency of occurrence to obtain a sorted candidate list, including:

[0029] The table name characters in the candidate table name list are arranged in descending order according to the frequency of occurrence of the table name, resulting in the sorted candidate table name list.

[0030] or,

[0031] The field names in the candidate field list are arranged in descending order according to the frequency of occurrence of the field, resulting in the sorted candidate field list.

[0032] Furthermore, the method also includes:

[0033] After the user completes the selection of the current table name character, the candidate table name list is obtained by arranging the accompanying table list corresponding to the current table name character and the frequency of the next table name character appearing when the current table name character appears in descending order.

[0034] Furthermore, the method also includes:

[0035] After the user completes the selection of the current field name, the candidate field list is obtained by arranging the accompanying field list corresponding to the current field name and the frequency of the next field name appearing when the current field name appears in descending order.

[0036] Secondly, embodiments of the present invention provide a data processing system for online task execution scripts, the system comprising:

[0037] The data collection module is used to collect online script execution task records;

[0038] The construction module is used to build the original database based on the online script execution task records;

[0039] The analysis module is configured to analyze the standard format data stored in the original database to obtain an analysis result, wherein the analysis result comprises character names, field frequency, table name frequency, inter-table relationship and inter-field relationship.

[0040] The statistics module is configured to count the analysis result to obtain a frequency and a companion list.

[0041] The acquisition module is configured to acquire the character names input by a user.

[0042] The retrieval module is configured to retrieve corresponding character names from the companion list according to the character names to generate a candidate list.

[0043] The arrangement module is configured to arrange the character names in the candidate list according to the frequency in descending order to obtain an arranged candidate list.

[0044] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program capable of running on the processor, and the processor implements the method described above when executing the computer program.

[0045] In a fourth aspect, an embodiment of the present application provides a computer readable medium having non-volatile program codes executable by a processor, wherein the program codes enable the processor to execute the method described above.

[0046] The data processing method and system for online task execution script provided by the embodiment of the present application comprise the following steps: collecting online script execution task records; constructing an original database according to the online script execution task records; analyzing the standard format data stored in the original database to obtain an analysis result, wherein the analysis result comprises character names, field frequency, table name frequency, inter-table relationship and inter-field relationship; counting the analysis result to obtain a frequency and a companion list; acquiring the character names input by a user; retrieving corresponding character names from the companion list according to the character names to generate a candidate list; arranging the character names in the candidate list according to the frequency in descending order to obtain an arranged candidate list; and filtering the appearance probability of table name characters and field name according to the analysis result to reduce the time for selecting the user in the prompt list and improve the user experience.

[0047] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art upon examination of the following or can be learned by practice of the present application. The objects and other advantages of the present application can be realized and attained by the structure particularly pointed out in the description, claims and drawings.

[0048] In order to make the above objectives, characteristics and advantages of the present application more apparent, comprehensible and easier to understand, the following preferred embodiments are specifically described in detail below, together with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to make the above objectives, characteristics and advantages of the present application more apparent, comprehensible and easier to understand, the following preferred embodiments are specifically described in detail below, together with the accompanying drawings.

[0050] Figure 1 A data processing method flow chart of an online task execution script provided by the embodiment one of the present application;

[0051] Figure 2 A data processing system schematic diagram of an online task execution script provided by the embodiment two of the present application.

[0052] ICON:

[0053] 1 - collection module; 2 - construction module; 3 - analysis module; 4 - statistics module; 5 - acquisition module; 6 - retrieval module; 7 - arrangement module. DETAILED DESCRIPTION

[0054] In order to make the above objectives, characteristics and advantages of the present application more apparent, comprehensible and easier to understand, the following preferred embodiments are specifically described in detail below, together with the accompanying drawings.

[0055] In order to make the above objectives, characteristics and advantages of the present application more apparent, comprehensible and easier to understand, the following preferred embodiments are specifically described in detail below, together with the accompanying drawings.

[0056] Embodiment one:

[0057] Figure 1 A data processing method flow chart of an online task execution script provided by the embodiment one of the present application;

[0058] Referring to Figure 1 The method comprises the following steps:

[0059] Step S101, collecting online script execution task records;

[0060] Step S102, constructing an original database according to the online script execution task records;

[0061] Step S103, the standard format data stored in the original database is parsed to obtain a parsing result, the parsing result including character name, field occurrence frequency, table name occurrence frequency, inter-table relationship and inter-field relationship;

[0062] Here, the standard format data stored in the original database is parsed to obtain a parsing result; the parsing result is stored in a standard format and an index is generated, and then stored in the original database.

[0063] Step S104, the parsing result is counted to obtain occurrence frequency and a companion list;

[0064] Step S105, a character name input by a user is obtained;

[0065] Step S106, the corresponding character name is retrieved from the companion list according to the character name, and an alternative list is generated;

[0066] Step S107, the character names in the alternative list are arranged in descending order according to the occurrence frequency to obtain an arranged alternative list.

[0067] In this embodiment, the table name characters and the field names are screened according to the occurrence probability based on the table name characters and the field names and the online script occurrence frequency and other characteristic information, which reduces the selection time of the user prompt list. The existing method is to directly sort the characters, and the user needs to select after screening.

[0068] Further, the online script execution task record includes a plurality of execution records, and step S102 includes the following steps:

[0069] Step S201, each of the plurality of execution records is marked to obtain a plurality of marked execution records;

[0070] Here, each execution record is marked, that is, each execution record is labeled with a corresponding label.

[0071] Step S202, the plurality of marked execution records are sorted to obtain original data in a standard format;

[0072] Step S203, the original data in the standard format is stored in a database to generate an original database.

[0073] Further, the occurrence frequency includes table name occurrence frequency and field occurrence frequency, and the companion list includes a companion table list and a companion field list; step S104 includes:

[0074] Step S301, the table name occurrence frequency is counted to obtain table name occurrence frequency;

[0075] Step S302, the field occurrence frequency is counted to obtain field occurrence frequency;

[0076] Step S303, according to the inter-table relationship, the frequency of other tables corresponding to the current table is counted, and the list of other tables is generated as the accompanying table list of the current table;

[0077] Step S304, according to the inter-field relationship, the frequency of other fields corresponding to the current field is counted, and the list of other fields is generated as the accompanying field list of the current field.

[0078] Further, the character name includes a table name character and a field name; step S106 includes:

[0079] According to the table name character, the corresponding table name character is retrieved from the accompanying table list, and an alternative table name list is generated;

[0080] Alternatively,

[0081] According to the field name, the corresponding field name is retrieved from the accompanying field list, and an alternative field list is generated.

[0082] Further, step S107 includes:

[0083] According to the table name frequency, the table name characters in the alternative table name list are arranged in descending order to obtain an arranged alternative table name list;

[0084] Alternatively,

[0085] According to the field frequency, the field names in the alternative field list are arranged in descending order to obtain an arranged alternative field list.

[0086] Specifically, according to the table name frequency, the table name characters in the alternative table name list are arranged in descending order, that is, the table name characters appearing earlier in the online script are arranged in front, and a prompt is given for user selection.

[0087] According to the field frequency, the field names in the alternative field list are arranged in descending order, that is, the field names appearing earlier in the online script are arranged in front, and a prompt is given for user selection.

[0088] Further, the method further includes the following steps:

[0089] Step S401, when the user completes the selection of the current table name character, according to the accompanying table list corresponding to the current table name character, and in combination with the frequency of the next table name character appearing when the current table name character appears, the alternative table name list is arranged in descending order to obtain an arranged alternative table name list;

[0090] Specifically, when multiple table associations occur, the above method is used to display the most likely appearing data table name at the forefront, and give a prompt for the user to select, and other table names are in turn.

[0091] When the user needs to enter the relationship of two tables, according to the accompanying relationship information between the tables stored in the original database, the most likely appearing accompanying relationship information in the online script is arranged in descending order, and a prompt is given for the user to select.

[0092] Step S402, when the user completes the selection of the current field name, the accompanying field list corresponding to the current field name is arranged in descending order according to the frequency of the next field name appearing when the current field name appears, and a sorted candidate field list is obtained.

[0093] Specifically, when the user inputs multiple field names, the above method can display the most likely appearing field name at the forefront, and give a prompt for the user to select, and other field names are in turn.

[0094] According to the online script execution record data, the table and the relationship between the tables are counted, the code prompt of the relationship between the tables is realized, and the existing big data development IDE technology does not have this function.

[0095] The application will be based on the online script execution task record, the execution frequency of the code of the script, the field use habit, the association frequency of the tables in the script, the association frequency of the fields and the tables, and the like, to sort and analyze, form a unique code automatic prompt strategy, thereby improving the development efficiency of the big data developers in the big data development IDE.

[0096] The data processing method of the online task execution script provided by the embodiment of the application comprises the following steps: collecting online script execution task records; constructing an original database according to the online script execution task records; parsing standard format data stored in the original database to obtain a parsing result, the parsing result comprising character names, field appearance frequencies, table appearance frequencies, table relationships and field relationships; counting the parsing result to obtain appearance frequencies and accompanying lists; obtaining a character name input by a user; retrieving corresponding character names from the accompanying lists according to the character name, and generating a candidate list; arranging the character names in the candidate list in descending order according to the appearance frequencies to obtain a sorted candidate list; and screening the appearance probabilities of table character names and field names according to the parsing result, reducing the time for the user to select in the prompt list, and improving user experience.

[0097] Embodiment two:

[0098] Figure 2 The data processing system schematic diagram of the online task execution script provided by the embodiment two of the application.

[0099] Referring to Figure 2 The system comprises:

[0100] A collection module 1 is configured to collect online script execution task records.

[0101] A construction module 2 is configured to construct an original database according to the online script execution task records.

[0102] A parsing module 3 is configured to parse standard format data stored in the original database to obtain a parsing result, the parsing result comprising character names, field occurrence frequencies, table occurrence frequencies, table-to-table relationships and field-to-field relationships.

[0103] A statistics module 4 is configured to perform statistics on the parsing result to obtain occurrence frequencies and accompanying lists.

[0104] An acquisition module 5 is configured to acquire character names input by a user.

[0105] A retrieval module 6 is configured to retrieve corresponding character names from the accompanying lists according to the character names and generate a candidate list.

[0106] An arrangement module 7 is configured to arrange the character names in the candidate list in descending order according to the occurrence frequencies to obtain an arranged candidate list.

[0107] The embodiment of the present application provides a data processing system for online task execution scripts, comprising: collecting online script execution task records; constructing an original database according to the online script execution task records; parsing standard format data stored in the original database to obtain a parsing result, the parsing result comprising character names, field occurrence frequencies, table occurrence frequencies, table-to-table relationships and field-to-field relationships; performing statistics on the parsing result to obtain occurrence frequencies and accompanying lists; acquiring character names input by a user; retrieving corresponding character names from the accompanying lists according to the character names and generating a candidate list; arranging the character names in the candidate list in descending order according to the occurrence frequencies to obtain an arranged candidate list; and filtering the occurrence probabilities of table name characters and field name characters according to the parsing result to reduce the time for the user to select in the prompt list and improve user experience.

[0108] The embodiment of the present application also provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data processing method for online task execution scripts provided by the above embodiment when executing the computer program.

[0109] The embodiment of the present application further provides a computer readable medium having non-volatile program codes executable by a processor, and the computer readable medium stores a computer program, and the computer program performs the steps of the data processing method of the online task execution script of the embodiment when executed by the processor.

[0110] The computer program product provided by the embodiment of the present application comprises a computer readable storage medium storing program codes, and the program codes comprise instructions for executing the method described in the foregoing method embodiment, and the specific implementation can be referred to the method embodiment, and will not be described here.

[0111] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the foregoing method embodiment, and will not be described here.

[0112] In addition, in the description of the embodiment of the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting" should be understood in a broad sense, for example, can be fixedly connected, or can be detachably connected, or integrally connected; can be mechanically connected, or can be electrically connected; can be directly connected, or can be indirectly connected through an intermediate medium; can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0113] If the functions are realized in the form of software function units and sold or used as independent products, the functions can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the part of the technical solutions which essentially contribute to the prior art or the part of the technical solutions can be embodied in the form of software product, and the computer software product is stored in a storage medium, and includes a plurality of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media which can store program codes.

[0114] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0115] Finally, it should be noted that the above-described embodiments are only specific implementations of the present application, which are used to illustrate the technical solutions of the present application, and are not limiting. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easily think of changes to the technical solutions recorded in the foregoing embodiments within the technical range disclosed by the present application, or make equivalent replacements to some technical features; and these modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data processing method of an online task execution script, characterized by, The method comprises: Collecting online script execution task records; Building an original database according to the online script execution task records; Pars ing standard format data stored in the original database to obtain a parsing result, the parsing result comprising character names, field occurrence frequencies, table name occurrence frequencies, table interrelation and field interrelation; Counting the parsing result to obtain occurrence frequencies and accompanying lists; Obtaining user input of the character names; Retrieving corresponding character names from the accompanying lists according to the character names and generating a candidate list; Arranging the character names in the candidate list in descending order according to the occurrence frequencies to obtain an arranged candidate list; The occurrence frequencies comprise table name occurrence frequencies and field occurrence frequencies, and the accompanying lists comprise accompanying table lists and accompanying field lists; counting the parsing result to obtain occurrence frequencies and accompanying lists comprises: Counting the table name occurrence frequencies to obtain the table name occurrence frequencies; Counting the field occurrence frequencies to obtain the field occurrence frequencies; According to the table interrelation, counting the frequencies of other tables corresponding to the current table when the current table appears, and generating a list of the other tables as the accompanying table list of the current table; According to the field interrelation, counting the frequencies of other fields corresponding to the current field when the current field appears, and generating a list of the other fields as the accompanying field list of the current field.

2. The data processing method of online task execution script according to claim 1, wherein, The online script execution task records comprise multiple execution records, and building an original database according to the online script execution task records comprises: Marking the multiple execution records respectively to obtain multiple marked execution records; Organizing the multiple marked execution records to obtain original data in a standard format; Storing the original data in a standard format into a database to generate the original database.

3. The data processing method of online task execution script according to claim 1, wherein, The character names comprise table name characters and field names; Retrieving corresponding character names from the accompanying lists according to the character names and generating a candidate list comprises: Retrieving corresponding table name characters from the accompanying table lists according to the table name characters and generating a candidate table name list; Or, Retrieving corresponding field names from the accompanying field lists according to the field names and generating a candidate field list.

4. The data processing method of online task execution script according to claim 3, characterized in that, Arranging the character names in the candidate list in descending order according to the occurrence frequencies to obtain an arranged candidate list comprises: Arranging table name characters in the candidate table name list in descending order according to the table name occurrence frequencies to obtain an arranged candidate table name list; Or, Arranging field names in the candidate field list in descending order according to the field occurrence frequencies to obtain an arranged candidate field list.

5. The data processing method of online task execution script according to claim 4, wherein, The method further comprises: When a user completes selection of a current table name character, arranging the accompanying table list corresponding to the current table name character in descending order according to the frequency of the next table name character appearing when the current table name character appears to obtain the arranged candidate table name list.

6. The data processing method of online task execution script according to claim 4, wherein, The method further comprises: When the user finishes selecting the current field name, a list of candidate field names is obtained according to the accompanying field list corresponding to the current field name and in descending order of frequency of occurrence of the next field name when the current field name occurs.

7. A data processing system for online task execution scripting, characterized by The system comprises: The acquisition module is configured to acquire online script execution task records. The construction module is configured to construct an original database according to the online script execution task records. The parsing module is configured to parse standard format data stored in the original database to obtain a parsing result, the parsing result comprising character names, field occurrence frequencies, table occurrence frequencies, table-to-table relationships and field-to-field relationships. The statistical module is configured to count the parsing result to obtain occurrence frequencies and accompanying lists. The acquisition module is configured to acquire a character name input by a user. The retrieval module is configured to retrieve a corresponding character name from the accompanying list according to the character name and generate a candidate list. The arrangement module is configured to arrange character names in the candidate list in descending order of occurrence frequencies to obtain an arranged candidate list. The occurrence frequencies comprise table occurrence frequencies and field occurrence frequencies, and the statistical module is specifically configured to: count the table occurrence frequencies to obtain the table occurrence frequencies; count the field occurrence frequencies to obtain the field occurrence frequencies; count frequencies of other tables corresponding to a current table according to the table-to-table relationships and generate a list of the other tables as an accompanying table list of the current table; count frequencies of other fields corresponding to a current field according to the field-to-field relationships and generate a list of the other fields as an accompanying field list of the current field.

8. An electronic device comprising a memory, a processor, the memory having stored thereon a computer program executable on the processor, characterized in that, The processor executes the computer program to implement the method of any one of claims 1 to 6.

9. A computer readable medium having a non-transitory program code executable by a processor, the program code comprising instructions for: The program code causes the processor to execute the method of any one of claims 1 to 6.

Images