Data acquisition method and device, electronic equipment and readable storage medium

Abstract

The application provides a data acquisition method and device, electronic equipment and a readable storage medium. The method comprises the following steps: acquiring a task execution file, a first task of the task execution file and a storage path, the task execution file comprising a latest execution serial number of the first task; acquiring a task address of the first task based on the storage path; and acquiring construction data corresponding to each execution serial number between the latest execution serial number and a reference execution serial number based on the task address, wherein the reference execution serial number is less than or equal to the latest execution serial number. The application can improve the efficiency of data acquisition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a data acquisition method and device, electronic equipment and a readable storage medium. BACKGROUND

[0002] When a development software is applied to various project development, a data report can be generated by analyzing the construction information of each task, so as to show the construction delivery efficiency of the product and improve the compilation efficiency. When the development software is used to execute tasks, the related task data is stored in the software server in the form of a file, and there is no fixed command to extract data. If the data needs to be extracted, the software page needs to be logged in to manually find and acquire the data, which is low in efficiency. SUMMARY

[0003] The present application provides a data acquisition method and device, electronic equipment and a readable storage medium to solve the problem of low data acquisition efficiency.

[0004] In a first aspect, an embodiment of the present application provides a data acquisition method, comprising:

[0005] acquiring a task execution file and a first task and a storage path of the task execution file, the task execution file comprising a latest execution serial number of the first task;

[0006] acquiring a task address of the first task based on the storage path;

[0007] acquiring construction data corresponding to each execution serial number between the latest execution serial number and a reference execution serial number based on the task address, wherein the reference execution serial number is less than or equal to the latest execution serial number.

[0008] In a second aspect, an embodiment of the present application further provides a data acquisition device, comprising:

[0009] a first acquisition module configured to acquire a task execution file and a first task and a storage path of the task execution file, the task execution file comprising a latest execution serial number of the first task;

[0010] a second acquisition module configured to acquire a task address of the first task based on the storage path;

[0011] a third acquisition module configured to acquire construction data corresponding to each execution serial number between the latest execution serial number and a reference execution serial number based on the task address, wherein the reference execution serial number is less than or equal to the latest execution serial number.

[0012] In a third aspect, the embodiments of the present application further provide an electronic device, comprising a processor, a memory, and a program or instructions stored in the memory and running on the processor, and the program or instructions are executed by the processor to implement the steps in the method according to the first aspect.

[0013] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium for storing a program, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in the method according to the first aspect.

[0014] In the embodiments of the present application, the task execution file and the first task and the storage path of the task execution file are obtained, the task address of the first task is obtained based on the storage path, and the construction data corresponding to each execution sequence number between the latest execution sequence number and the reference execution sequence number is obtained based on the task address. The construction data of the task can be quickly obtained through the task address and the execution sequence number, and the efficiency of obtaining the construction data is improved. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] Figure 1 is a flowchart of the data acquisition method provided by the embodiments of the present application;

[0017] Figure 2 is a schematic diagram of dynamic updating of the number of Jenkins task executions provided by the embodiments of the present application;

[0018] Figure 3 is one of the flowcharts of the construction log acquisition method provided by the embodiments of the present application;

[0019] Figure 4 is another flowchart of the construction log acquisition method provided by the embodiments of the present application;

[0020] Figure 5 is one of the structural diagrams of the data acquisition device provided by the embodiments of the present application;

[0021] Figure 6 is another structural diagram of the data acquisition device provided by the embodiments of the present application;

[0022] Figure 7 is a third structural diagram of the data acquisition device provided by the embodiments of the present application;

[0023] Figure 8 FIG. 4 is a structural schematic diagram of a data acquisition device according to an embodiment of the present application;

[0024] Figure 9 FIG. 5 is a structural schematic diagram of a data acquisition device according to an embodiment of the present application;

[0025] Figure 10 FIG. 6 is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0027] The terms "first", "second", and the like in the embodiments of the present application are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, a method, a system, a product, or an apparatus that includes a series of steps or units does not necessarily limit to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to the process, the method, the product, or the apparatus. In addition, "and / or" is used in the present application to represent at least one of the connected objects, for example, A and / or B and / or C represents 7 cases including A alone, B alone, C alone, A and B both exist, B and C both exist, A and C both exist, and A, B and C all exist.

[0028] Please refer to Figure 1 , Figure 1 FIG. 1 is a flowchart of a data acquisition method according to an embodiment of the present application, as shown in the figure, including the following steps: Figure 1

[0029] Step 101, acquiring a task execution file and a first task and a storage path of the task execution file, the task execution file including a latest execution serial number of the first task.

[0030] Among them, the above-mentioned task execution file can be one-to-one corresponding to the above-mentioned first task, for example: when acquiring multiple task execution files, the first task and the storage path of each task execution file can be acquired, the storage paths of different tasks are different, and each task execution file contains the latest execution serial number of the corresponding first task.

[0031] ​Specifically, regardless of the creation method and creation time of the first task, each execution of the first task will update the latest execution sequence number in the task execution file, so that the task execution file records the latest execution sequence number corresponding to the latest execution count of the first task.

[0032] Step 102: Obtain the task address of the first task based on the storage path.

[0033] Step 103: Based on the task address, obtain the construction data corresponding to each execution sequence number between the latest execution sequence number and the reference execution sequence number, wherein the reference execution sequence number is less than or equal to the latest execution sequence number.

[0034] Each task has an execution sequence number that corresponds to a set of construction data for that task. Therefore, based on the task address and execution sequence number, the construction data corresponding to each execution sequence number of the first task can be determined.

[0035] The reference execution sequence number can be determined based on the number of times the task execution file is retrieved. For example, if the task execution file is being retrieved for the first time, the reference execution sequence number can be set to the minimum execution sequence number 1, thereby retrieving the build data corresponding to all execution sequences of the first task, thus obtaining all historical build data of the first task up to this point. If the task execution file is not being retrieved for the first time, the latest execution sequence number of the previously retrieved task execution file can be queried. Then, the reference execution sequence number can be the next execution sequence number after the latest execution sequence number of the previously retrieved task execution file. Assuming that the task execution file for task A is being retrieved for the second time, and the latest execution sequence number of task A retrieved this time is 4, while the latest execution sequence number of task A retrieved last time was 2, then the reference execution sequence number is 3, and the build data retrieved this time is the build data corresponding to the latest execution sequence number 4 and execution sequence number 3. In addition, if the previous attempt to obtain the above build data failed, it can be assumed that the previous attempt did not exist. Taking task A as an example again, if the previous attempt to obtain the build data of task A failed, the current attempt to obtain the build data can be understood as the first attempt, that is, the reference execution sequence number is 1. The build data obtained this time includes the build data corresponding to the latest execution sequence number 4, execution sequence number 3, execution sequence number 2 and execution sequence number 1.

[0036] In this embodiment, a task execution file, a first task of the task execution file, and a storage path are obtained. The task address of the first task is obtained based on the storage path. The construction data corresponding to each execution sequence number between the latest execution sequence number and the reference execution sequence number is obtained based on the task address. The construction data of the task can be quickly obtained through the task address and execution sequence number, thereby improving the efficiency of obtaining the construction data.

[0037] And, based on the task address, the construction data corresponding to each execution sequence number between the latest execution sequence number and the reference execution sequence number is obtained, so that the construction data corresponding to a segment of execution sequence numbers between the latest execution sequence number and the reference execution sequence number can be directly obtained, that is, the history construction data of each execution of the first task can be obtained through the task execution file, and the completeness of the construction data obtained is improved.

[0038] Optionally, the reference execution sequence number is the next execution sequence number of the maximum execution sequence number of the first task stored in the database server.

[0039] After the step 103 of obtaining the construction data corresponding to each execution sequence number between the latest execution sequence number and the reference execution sequence number based on the task address, the method can further include the following steps:

[0040] The construction log is sent to the database server, and the construction log includes the task address, the construction data, and the execution sequence number of the construction data.

[0041] In the prior art, when the construction data needs to be counted, the construction data of each task execution can be collected in real time when each task is newly added, but it is difficult to count the history construction data of the task which has been created and executed multiple times before counting.

[0042] The database server can be used to store the construction log, and the construction data corresponding to each execution sequence number between the latest execution sequence number and the reference execution sequence number can be understood as newly added history construction data. The newly added history construction data is sent to the database server, so that the storage and counting of the history construction data can be realized.

[0043] The steps 101 to 103 can be repeatedly executed at a preset time interval, so that even if the data acquisition fails, the construction data can be obtained at the next execution. The step 103 can obtain the construction data corresponding to each execution sequence number between the latest execution sequence number and the reference execution sequence number, that is, all newly added construction data. The preset time interval can be customized as every hour, every day, every week, etc. For example, if the real-time performance of the construction data obtained is to be improved, the preset time interval can be shortened.

[0044] In the embodiment, the construction log is sent to the database server, and the construction log can be stored in the database server, so that the construction log can be collected, counted, etc.

[0045] Optionally, the construction log is sent to the database server, and the construction log can be stored in the database server, so that the construction log can be collected, counted, etc.

[0046] determining, based on the task address and the execution sequence number of the build data, that the build log does not exist in the database server, and sending the build log to the database server.

[0047] It can be understood that the task address and the execution sequence number of the build data can be combined to determine a certain execution number of a certain task, that is, the task address and the execution sequence number of the build data of the build log can be used to determine whether the build log has been stored in the database server, and if the build log has been stored in the database server, data duplication can be avoided; if the build log does not exist in the database server, the newly added build log can be stored.

[0048] In this embodiment, based on the task address and the execution sequence number of the build data, the task corresponding to the build log and the execution number thereof can be determined, so as to determine whether the build log exists in the database server, and in the case that the build log does not exist in the database server, the build log is sent to the database server, so that while storing all newly added build data, data duplication stored in the database server can be avoided.

[0049] Optionally, the step 103 of acquiring the build data corresponding to each execution sequence number between the latest execution sequence number and the reference execution sequence number based on the task address can specifically include:

[0050] acquiring the reference execution sequence number;

[0051] comparing the size of each execution sequence number between the latest execution sequence number and the minimum execution sequence number with the reference execution sequence number in sequence;

[0052] acquiring the build data corresponding to each execution sequence number greater than or equal to the reference execution sequence number based on the task address in sequence.

[0053] The minimum execution sequence number can be understood as the execution sequence number of the first execution of the first task, and the build data corresponding to each execution sequence number between the latest execution sequence number and the minimum execution sequence number is acquired in sequence, that is, the build data can be acquired in sequence from the back to the front according to the execution sequence number, if the build data corresponding to the reference execution sequence number indicates that all newly added build data of the first task has been acquired, there is no need to acquire the build data corresponding to the execution sequence number before the reference execution sequence number, and there is no need to compare the build data corresponding to the execution sequence number less than the reference execution sequence number one by one to determine whether it is stored in the database server, which facilitates the acquisition of the newly added build data.

[0054] In this embodiment, by sequentially comparing the size of each execution sequence number between the latest execution sequence number and the minimum execution sequence number with the reference execution sequence number, the construction data corresponding to each execution sequence number newly added by the first task can be obtained, so that the construction data newly added by the first task relative to the last time can be directly obtained, and the omission of construction data caused by the failure of data acquisition can be reduced.

[0055] Optionally, after the construction log is sent to the database server, the method can further include the following steps:

[0056] receiving a data display instruction;

[0057] obtaining the construction log stored in the database server based on the data display instruction, and displaying the construction log.

[0058] In this embodiment, the construction log stored in the database server is obtained based on the data display instruction, and the construction log is displayed, so that the data in the database can be pulled at any time according to the display requirement, and a corresponding display report can be generated and displayed.

[0059] The various optional embodiments introduced in the embodiments of the present application can be combined with each other to be implemented without conflict, or can be implemented alone, and the embodiments of the present application do not limit this.

[0060] For the convenience of understanding, specific examples are as follows:

[0061] Jenkins is a popular continuous integration / continuous delivery (CI / CD) general tool in the software industry, and is widely used in various projects. By analyzing the construction information of each task of Jenkins, a data report can be generated, which can well show the construction delivery rate of the product, improve the compilation efficiency, and infer the potential problem link in the CI / CD process.

[0062] The embodiments of the present application provide a construction log acquisition method, as shown in Figures 2 to 4 As shown, the task address of all Jenkins tasks and all execution sequence numbers of each task can be obtained based on the "nextBuildNumber" file stored in the Jenkins server, and the construction log information of the execution sequence numbers corresponding to all tasks can be parsed through the Jenkins interface.

[0063] As shown in Figure 3 and Figure 4 As shown, first, a data collection task (jenkins_job_statistic) is created in Jenkins, and the timing interval can be customized as every hour, every day, every week, etc.,Figure 3 is a flowchart of a process for first performing a timing data collection task (jenkins_job_statistic), and Figure 4 is a flowchart of a process for restarting the timing data collection task (jenkins_job_statistic) after a timing interval. The execution of the timing data collection task (jenkins_job_statistic) can specifically include the following steps:

[0064] Step 1: Search for the "nextBuildNumber" file under the JENKINS_HOME directory of the Jenkins server, and get the file path of each "nextBuildNumber" file. After uniform string trimming, replacement, and Jenkins master address splicing, the task address of all Jenkins tasks can be obtained.

[0065] Among them, the "nextBuildNumber" file is automatically generated after the first execution of each Jenkins task, and the "nextBuildNumber" file corresponds to the Jenkins task one by one, and is used to record the execution serial number needed for the next execution task. The content of this file will be dynamically updated with the execution times of the Jenkins task, that is, the execution serial number recorded by the "nextBuildNumber" file is the latest execution times of the corresponding Jenkins task plus 1.

[0066] Step 2: Take a certain Jenkins task A as an example, and get the latest execution serial number of task A by subtracting one from the execution serial number recorded in the "nextBuildNumber" file corresponding to task A.

[0067] Step 3: Pass the parameters: the task address of task A and the corresponding latest execution serial number to the API (Application Programming Interface) interface of Jenkins. Through the API interface, the latest execution build log information of task A can be parsed, including but not limited to: the execution result, execution duration, execution start time, etc. of the task. Insert the build log information, task address and latest execution serial number of task A into the database as the build log of task A.

[0068] Step 4: Loop step 3, and subtract one from the execution serial number each time until the serial number is 1. All historical build information of task A can be obtained so far, and all historical build information is inserted into the database for storage.

[0069] Step 5: As Figure 4As shown, when the timing data collection task (jenkins_job_statistic) is started again, the task A can be executed multiple times in the timing interval, and the corresponding "nextBuildNumber" file content is automatically updated to the latest execution sequence number plus one. At this time, according to step 3, the build log information corresponding to the latest execution sequence number is parsed, and the database is queried to determine whether the build log information has been inserted into the database (the task address + execution sequence number of each build log information is unique). If not, the build information is inserted, the sequence number is reduced by one, and step 3 is continued. If yes, the insertion is ended. In this way, the new build information of the task A can be inserted into the database.

[0070] Step 6: According to the other task addresses and the latest execution sequence numbers of the corresponding tasks obtained in steps 1 and 2, the task A is executed in the same way one by one, and all historical build information of each task of Jenkins can be collected.

[0071] Among them, all tasks can be executed again after the timing interval, and the next execution sequence number in the corresponding "nextBuildNumber" file will be automatically updated. The build log corresponding to the execution sequence number between the latest execution sequence number and the next execution sequence number in the "nextBuildNumber" file is the new build log. In this way, after the timing data collection task (jenkins_job_statistic) is executed, all new build log information of all tasks will be inserted into the database. Even if there is a problem with the network or database connection during the execution of the timing task, according to the execution method of step 5, all new build information will be updated to the database after the next timing data collection task (jenkins_job_statistic) is executed.

[0072] In the embodiment of the application, the build log information data can be pulled from the database at any time to generate a display report and display according to the needs of the performance platform front end display.

[0073] In this way, in the embodiment of the application, the task address of all tasks of Jenkins and all execution sequence numbers corresponding to each task can be obtained by searching and processing the "nextBuildNumber" file, and all build logs of the corresponding task can be obtained according to the task address and the execution sequence number.

[0074] In addition, the timing data acquisition manner in the embodiment of the application does not depend on the creation manner or creation time of the Jenkins task, and even before the creation of the timing data acquisition task (jenkins_job_statistic), the latest execution sequence number of each task and the execution sequence number of the last successful insertion into the database can be obtained through the timing data acquisition manner, all the newly added build data between the two execution sequence numbers can be inserted into the database, and the non-real-time piece-by-piece insertion can be limited to the time of starting data acquisition, and all the build data of all the tasks can be counted, the individual data loss caused by network or database connection problems during real-time data acquisition can be effectively avoided, and all the historical build logs of all the tasks can be inserted into the database through the execution of the timing data acquisition task.

[0075] Please refer to Figure 5 , Figure 5 is one of the structural schematic diagrams of the voice awakening device provided in the embodiment of the application, as shown in Figure 5 The data acquisition device 500 comprises:

[0076] The first acquisition module 501 is configured to acquire a task execution file, a first task of the task execution file and a storage path, and the task execution file comprises a latest execution sequence number of the first task.

[0077] The second acquisition module 502 is configured to acquire a task address of the first task based on the storage path.

[0078] The third acquisition module 503 is configured to acquire build data corresponding to each execution sequence number between the latest execution sequence number and a reference execution sequence number based on the task address, and the reference execution sequence number is less than or equal to the latest execution sequence number.

[0079] Optionally, as shown in Figure 6 The reference execution sequence number is the next execution sequence number of the maximum execution sequence number of the first task stored in the database server.

[0080] The data acquisition device 500 can further comprise:

[0081] The sending module 504 is configured to send a build log to the database server, and the build log comprises the task address, the build data and the execution sequence number of the build data.

[0082] Optionally, as shown in Figure 7 The sending module 504 can specifically comprise:

[0083] The sending unit 5041 is configured to send the build log to the database server in a case where the database server does not exist the build log based on the task address and the execution sequence number of the build data.

[0084] Optionally, as shown in Figure 8 the third acquisition module 503 can include:

[0085] The first acquisition unit 5031 is configured to acquire the reference execution sequence number.

[0086] The comparison unit 5032 is configured to compare sizes of each execution sequence number between the latest execution sequence number and the minimum execution sequence number and the reference execution sequence number in sequence.

[0087] The second acquisition unit 5032 is configured to acquire build data corresponding to each execution sequence number greater than or equal to the reference execution sequence number in sequence based on the task address.

[0088] Optionally, as shown in Figure 9 the data acquisition apparatus 500 can further include:

[0089] The receiving module 505 is configured to receive a data display instruction.

[0090] The display module 506 is configured to acquire the build log stored in the database server based on the data display instruction, and display the build log.

[0091] The data acquisition apparatus 500 can realize all the processes of the method embodiments of the present application Figure 1 and achieve the same beneficial effects. To avoid repetition, details are not described herein.

[0092] The present application also provides an electronic device. Please refer to Figure 10 The electronic device 1000 includes a processor 1001, a memory 1002, a program or instruction stored in the memory 1002 and executable on the processor 1001. The program or instruction is executed by the processor 1001 to realize all the processes of the image recognition method embodiments described above and achieve the same technical effects. To avoid repetition, details are not described herein.

[0093] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiment methods can be completed by program instruction related hardware, and the program can be stored in a readable medium.

[0094] The present application also provides a computer readable storage medium for storing a program, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to realize the above-mentionedFigure 1 Any step in the corresponding method embodiments and with the same technical effects will not be repeated here.

[0095] The storage medium, such as a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0096] The above is the preferred embodiment of the application, it should be pointed out that, for those skilled in the art, without departing from the principles described in this application, can make a number of improvements and refinements, these improvements and refinements should also be considered as the protection scope of the application.

Claims

1. A data acquisition method, characterized by, The method comprises the following steps: obtain a task execution file and a first task and a storage path of the task execution file, the task execution file comprising a latest execution sequence number of the first task; obtain a task address of the first task based on the storage path; obtain construction data corresponding to each execution sequence number between the latest execution sequence number and a reference execution sequence number based on the task address, wherein the reference execution sequence number is less than or equal to the latest execution sequence number; the reference execution sequence number is the next execution sequence number of the maximum execution sequence number of the first task stored in the database server; the step of obtaining the construction data corresponding to each execution sequence number between the latest execution sequence number and a reference execution sequence number based on the task address comprises the following steps: obtain the reference execution sequence number; compare the size of each execution sequence number between the latest execution sequence number and the minimum execution sequence number with the reference execution sequence number in sequence; obtain the construction data corresponding to each execution sequence number greater than or equal to the reference execution sequence number based on the task address in sequence; after the step of obtaining the construction data corresponding to each execution sequence number between the latest execution sequence number and a reference execution sequence number based on the task address, the method further comprises the following steps: send a construction log to the database server, the construction log comprising the task address, the construction data, and the execution sequence number of the construction data; the step of sending the construction log to the database server comprises the following steps: determine whether the database server does not store the construction log based on the task address and the execution sequence number of the construction data, and send the construction log to the database server if it is determined that the database server does not store the construction log.

2. The method of claim 1, wherein, after the step of sending the construction log to the database server, the method further comprises the following steps: receive a data display instruction; obtain the construction log stored in the database server based on the data display instruction, and display the construction log.

3. A data acquisition device, characterized by The method comprises the following steps: a first obtaining module is configured to obtain a task execution file and a first task and a storage path of the task execution file, the task execution file comprising a latest execution sequence number of the first task; a second obtaining module is configured to obtain a task address of the first task based on the storage path; a third obtaining module is configured to obtain construction data corresponding to each execution sequence number between the latest execution sequence number and a reference execution sequence number based on the task address, wherein the reference execution sequence number is less than or equal to the latest execution sequence number; the reference execution sequence number is the next execution sequence number of the maximum execution sequence number of the first task stored in the database server; the third obtaining module comprises the following steps: a first obtaining unit is configured to obtain the reference execution sequence number; a comparison unit is configured to compare the size of each execution sequence number between the latest execution sequence number and the minimum execution sequence number with the reference execution sequence number in sequence; a second obtaining unit is configured to obtain the construction data corresponding to each execution sequence number greater than or equal to the reference execution sequence number based on the task address in sequence; a sending module is configured to send a construction log to the database server, the construction log comprising the task address, the construction data, and the execution sequence number of the construction data; The sending module comprises: The sending unit is configured to determine, based on the task address and the execution sequence number of the build data, that the build log is not stored in the database server, and send the build log to the database server.

4. An electronic device, comprising: The processor, the memory, and a program or instructions stored on the memory and running on the processor, the program or instructions being executed by the processor to implement the steps of the data acquisition method according to any one of claims 1 to 2.

5. A computer readable storage medium, characterized in that, A computer readable storage medium for storing a program, the computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the steps in the data acquisition method according to any one of claims 1 to 2.

Images