Data acquisition method, electronic equipment and storage medium

Abstract

The invention provides a data acquisition method, electronic equipment and a storage medium. The application program data collection method comprises the steps of deploying a to-be-collected application program in a containerization mode to obtain a container mirror image corresponding to the to-be-collected application program; executing data acquisition operation on the container mirror image to obtain observable data of the to-be-acquired application program; wherein the data acquisition operation comprises the following steps: starting the container mirror image; and performing data acquisition on the container mirror image by utilizing a data acquisition program to obtain observable data of the application program to be acquired. According to the method, the deployment period in the data acquisition process can be simplified, the acquisition efficiency is improved, and a user can conveniently verify the acquisition effect of the data acquisition program.

Description

Technical Field

[0001] The present invention relates to the technical field of data acquisition, and more specifically to a data acquisition method, electronic equipment and storage medium. Background Art

[0002] In today's rapidly developing digital age, the complexity and dynamism of applications are increasing, making the need for application observability particularly urgent. In order to achieve efficient application management and operation and maintenance, it is usually necessary to collect observable data of applications to facilitate the management and maintenance of applications.

[0003] In order to ensure the accuracy of observable data collection, it is usually necessary to verify the collection effect of the data collection program. In the related art, the application to be collected is usually started manually, and then the data collection of the application to be collected is completed through the data collection program, and the collection effect of the data collection program is verified based on the collected data. This method is inefficient.

[0004] In view of this, the present invention is proposed. Summary of the invention

[0005] The present invention is proposed in view of the above problems. According to one aspect of the present invention, a method for collecting data of an application is provided, comprising: Deploy the application to be collected in a containerized manner to obtain a container image corresponding to the application to be collected; Performing a data collection operation on the container image to obtain observable data of the application to be collected; The data collection operation includes: Starting the container image; The data collection program is used to collect data from the container image to obtain observable data of the application program to be collected.

[0006] Exemplarily, the number of the to-be-collected applications is multiple; the multiple to-be-collected applications have different application information, and the application information includes at least one of the following information: version, deployment form, and usage form; The step of deploying the application to be collected in a containerized manner to obtain a container image corresponding to the application to be collected includes: Deploy each of the multiple applications to be collected in a containerized manner to obtain container images corresponding to the multiple applications to be collected one by one; The performing a data collection operation on the container image to obtain observable data of the application to be collected includes: The data collection operation is performed on each container image corresponding to the multiple application programs to be collected, so as to obtain observable data of each of the multiple application programs to be collected.

[0007] Exemplarily, the performing the data collection operation on each container image corresponding to the multiple application programs to be collected includes: Performing the data collection operation on each container image corresponding to the multiple application programs to be collected in sequence; After obtaining the observable data of each of the plurality of applications to be collected, the method further includes: A test report is generated based on the observable data of each of the multiple application programs to be collected.

[0008] Exemplarily, before using the data collection program to collect data from the container image, the data collection operation further includes: The sample data is imported into the container image, where the sample data is working data of the application to be collected in a real working environment.

[0009] Exemplarily, after starting the container image, the data collection operation further includes: The startup parameters of the corresponding application to be collected are set in the container image.

[0010] Exemplarily, the step of performing data collection operations on the container image is performed according to a preset cycle, or the step of performing data collection operations on the container image is performed when the application to be collected changes, or the step of performing data collection operations on the container image is performed when the data collection program changes.

[0011] Exemplarily, after collecting data from the container image using a data collection program, the data collection operation further includes: Perform integrity check on the collected observable data.

[0012] Exemplarily, after collecting data from the container image using a data collection program, the data collection operation further includes: The data type of the collected observable data is compared with the expected type to determine whether the data type of the collected observable data meets expectations.

[0013] According to another aspect of the present invention, there is provided an electronic device, including a processor and a memory, wherein a computer program is stored in the memory, and the processor is used to execute the computer program to implement the above method.

[0014] According to another aspect of the present invention, a computer-readable storage medium is provided, storing a computer program / instruction, and the computer program / instruction implements the above method when executed by a processor.

[0015] In the above technical solution, data is collected from the container image of the application to be collected to obtain observable data of the application to be collected. This solution can simplify the deployment cycle in the data collection process. At the same time, this solution can realize automatic data collection, replacing the manual collection process in traditional technology, which helps to improve the collection efficiency and facilitates users to verify the collection effect of the data collection program.

[0016] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0018] Figure 1 A schematic flow chart showing a data collection method according to an embodiment of the present invention; Figure 2 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

[0020] As described above, in the related art, the application to be collected is usually started manually by manual means, and then the data collection of the application to be collected is completed through the data collection program. Since each change of the data collection program may cause differences in the collection behavior, each change needs to be tested manually. This process may be time-consuming, especially when there are a large number of applications to be collected. Each change of the data collection program must repeat this process, resulting in an extension of the overall deployment cycle and increased time and labor costs. In view of this, the present invention provides a data collection method, an electronic device, and a storage medium. The method can simplify the deployment cycle in the data collection process, improve the collection efficiency, and facilitate users to verify the collection effect of the data collection program.

[0021] According to one aspect of an embodiment of the present invention, a method for collecting data of an application is provided. Figure 1 FIG. 2 is a schematic flow chart of a data collection method according to an embodiment of the present invention. Figure 1 As shown, the data collection method may include the following steps S110 and S120.

[0022] In step S110, the application to be collected is deployed in a containerized manner to obtain a container image corresponding to the application to be collected.

[0023] In this article, the applications to be collected include but are not limited to database systems (such as MySQL), web servers, any backend services, caches (such as Redis / MemcacheD), operating systems (Linux / Windows / macOS), and any other applications that need to collect observable data.

[0024] In the solution of this example, the application to be collected can be deployed in a containerized manner. For example, the application to be collected can be deployed in a Docker manner. Of course, the application to be collected can also be deployed using containerization technologies such as containerd, which will not be described in detail.

[0025] In step S120, a data collection operation is performed on the container image to obtain observable data of the application to be collected.

[0026] The data collection operation may include the following steps S121 and S124.

[0027] In step S121, the container image is started.

[0028] In step S124, data collection is performed on the container image using a data collection program to obtain observable data of the application program to be collected.

[0029] In the scheme of this example, when performing data collection operations on a container image, the container image can be started first, and then data can be collected using a data collection program. For example, the container image is an image obtained through Docker (can be referred to as a Docker image), and the application to be collected is MySQL. In this case, the Docker image can be regarded as a MySQL instance. In this embodiment, the Docker API can be called first to start the Docker image. After the Docker image is started, the data collection program is started, and the data during the operation of the Docker image is collected using the data collection program, thereby obtaining observable data of the application to be collected.

[0030] In the above technical solution, data is collected from the container image of the application to be collected to obtain observable data of the application to be collected. This solution can simplify the deployment cycle in the data collection process. At the same time, this solution can realize automatic data collection, replacing the manual collection process in traditional technology, which helps to improve the collection efficiency, and can facilitate users to verify the collection effect of the data collection program, and realize high-accuracy rapid verification.

[0031] Exemplarily, there are multiple applications to be collected; the multiple applications to be collected have different application information, and the application information includes at least one of the following information: version, deployment form, and usage form.

[0032] Step S110, deploying the application to be collected in a containerized manner to obtain a container image corresponding to the application to be collected, can specifically include the following steps: deploying each of the multiple applications to be collected in a containerized manner to obtain container images corresponding one-to-one to the multiple applications to be collected.

[0033] Step S124, performing a data collection operation on the container image to obtain observable data of the application to be collected, may specifically include the following steps: performing a data collection operation on each container image corresponding to the multiple applications to be collected, respectively, to obtain observable data of each of the multiple applications to be collected.

[0034] It can be understood that the data collection program usually needs to support data collection of different versions, different deployment forms, and different usage forms of the same application. In the solution of this example, multiple applications to be collected with different application information are the same application with different versions, different deployment forms, and different usage forms. Taking MySQL as an example, multiple applications to be collected can be multiple different MySQL versions and release variants (adjusted according to different deployment forms or usage forms). In this case, multiple MySQL container images can be obtained in a containerized manner, so that data can be collected for each application to be collected through the data collection program in subsequent operations, so that the data collection effect of the data collection program on each application to be collected can be verified based on the data collection results.

[0035] In the solution of this example, after obtaining multiple container images, data collection operations can be performed on each container image corresponding to multiple application programs to be collected. For example, a data collection program can be used to collect data for each container image at the same time. Of course, data collection can also be performed in sequence according to a preset order, which will not be repeated.

[0036] The above technical solution can realize automated data collection for different versions, different deployment forms, and different usage forms of the same application, thereby helping to quickly obtain observable data and facilitating users to verify the data collection program's support for the application through observable data.

[0037] Exemplarily, performing the data collection operation on each container image corresponding to the multiple application programs to be collected includes: performing the data collection operation on each container image corresponding to the multiple application programs to be collected in sequence.

[0038] After obtaining the observable data of each of the multiple applications to be collected, the method further includes: generating a test report according to the observable data of each of the multiple applications to be collected.

[0039] In this example, consider performing data collection operations on each container image in turn. This is to reduce resource consumption during data collection and avoid interference between multiple collection threads. In this case, the collected data to be observed can better measure the data collection effect of the data collection program on the application.

[0040] After the observable data of multiple applications to be collected are collected, the observable data of each application to be collected can be counted to obtain a complete test report. Taking MySQL as an example, after completing the observable data of different MySQL versions and releases, a complete test report on MySQL can be obtained. This test report can provide users with a more intuitive basis for evaluating the collection effect, which helps to improve the user experience.

[0041] Exemplarily, before using the data collection program to collect data from the container image, the data collection operation further includes: step S123, importing sample data into the container image, the sample data being working data of the application to be collected in a real working environment.

[0042] In the solution of this example, after the container image is started, consider importing the working data of the application to be collected in the real working environment into the container image. This data can make the container image closer to the normally running application instance. Taking MySQL as an example, after MySQL is successfully started, some application data can be injected through the MySQL SDK to make the container image look like a normally running MySQL instance. In this way, when using the data collection program to collect observable data, the collected data will be closer to the effect of real production. The data collected in this way can more accurately evaluate the collection effect of the data collection program.

[0043] Exemplarily, after starting the container image, the data collection operation further includes: step S122, setting the corresponding startup parameters of the application to be collected in the container image.

[0044] The startup parameters may include but are not limited to user name, password, etc.

[0045] It is understandable that the behavior of the application will be different with different startup parameters. The purpose here is to expect the data collection program to reflect these differences in the final collected data, so as to verify the correctness of the collection program.

[0046] Exemplarily, the step of performing data collection operations on the container image is performed according to a preset cycle, or the step of performing data collection operations on the container image is performed when the application to be collected changes, or the step of performing data collection operations on the container image is performed when the data collection program changes.

[0047] In some embodiments, the step of performing data collection operations on the container image is performed according to a preset period. The preset period is selected according to actual needs. For example, it can be deployed and run in Nightly mode.

[0048] In some embodiments, the step of performing data collection operations on the container image is performed when the application to be collected changes. It is understood that the change in the application to be collected may be a change in the number of applications to be collected, or a change in the version of some of the multiple applications to be collected.

[0049] In some embodiments, the step of performing the data collection operation on the container image is performed when the data collection program changes, wherein the change in the data collection program may be a change in the version of the data collection program.

[0050] The above technical solution can achieve long-term and repeated automatic testing by controlling the steps of performing data collection operations on container images to be executed periodically or according to preset conditions (for example, changes in the application to be collected or changes in the data collection program), which helps to improve the test intensity and robustness.

[0051] Exemplarily, after collecting data from the container image using the data collection program, the data collection operation further includes: performing integrity verification on the collected observable data.

[0052] In some embodiments, the test report also includes integrity verification results corresponding to each container image.

[0053] In this example, after collecting data from the container image, the collected observable data can be integrity checked to verify whether the current collection process has collected all the data that needs to be collected. The integrity check result can help users more accurately determine the collection effect of the data collection software.

[0054] Exemplarily, after collecting data from the container image using the data collection program, the data collection operation further includes: comparing the data type of the collected observable data with the expected type to determine whether the data type of the collected observable data meets expectations.

[0055] In some embodiments, the test report also includes a type comparison result corresponding to each container image.

[0056] In this example, after collecting data from the container image, you can further verify whether the data type (int / float / string, etc.) meets expectations. The verification result can help users more accurately determine the collection effect of the data collection software.

[0057] The following is an introduction to the implementation of the present invention through a specific embodiment. In this embodiment, each application to be collected can be deployed in a Docker manner, so as to obtain a Docker image corresponding to each application to be collected. When it is necessary to verify the collection effect of the data collection program, the Docker API can be called first to start the Docker image and set the startup parameters of MySQL. After MySQL is successfully started, some application data is injected through the MySQL SDK to make the container image look like a normally running MySQL instance. Then, the data collection program is started to collect data from the Docker image. After the data collection is completed, the collected observable data is integrity checked to verify whether the current collection process has collected all the data that needs to be collected. At the same time, the data type of the collected observable data can be compared with the expected type to determine whether the data type of the collected observable data meets expectations. After the observable data collection of a single MySQL is completed, data can be collected for other MySQL versions and release variants through the above operations. Finally, a complete test report on MySQL is obtained based on the collection results. This solution can simulate the collected object (i.e. the application to be collected) through Docker. It can be applied in the CI / CD environment of the collection program, or run locally by the developer. It only requires a Docker environment and the preparation of the Docker image of the collected object.

[0058] According to yet another aspect of the embodiments of the present invention, an electronic device is provided. Figure 2 FIG. 2 shows a schematic block diagram of an electronic device according to an embodiment of the present invention. Figure 2 As shown, the electronic device 200 includes: a processor 210 and a memory 220. The memory 220 stores a computer program, and the processor 210 is used to execute the computer program to implement the above method.

[0059] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided. The storage medium stores a computer program / instruction, and the computer program / instruction implements the above method when executed by a processor. The storage medium may include, for example, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0060] A person skilled in the art can easily understand the implementation structure, working principle and beneficial effects of the electronic device and the computer-readable storage medium by reading the above method, and will not be described in detail for the sake of brevity.

[0061] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

[0062] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0063] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0064] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0065] Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0066] Those skilled in the art will understand that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstract and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise explicitly stated, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0067] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0068] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all functions of some modules in an electronic device according to an embodiment of the present invention. The present invention may also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0069] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0070] The above is only a specific embodiment of the present invention or an explanation of a specific embodiment. The protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A data collection method for an application, characterized in that: include: Deploy the application to be collected in a containerized manner to obtain a container image corresponding to the application to be collected; Performing a data collection operation on the container image to obtain observable data of the application to be collected; The data collection operation includes: Starting the container image; The data collection program is used to collect data from the container image to obtain observable data of the application program to be collected.

2. The data collection method according to claim 1, characterized in that: The number of the applications to be collected is multiple; the multiple applications to be collected have different application information, and the application information includes at least one of the following information: version, deployment form, and usage form; The step of deploying the application to be collected in a containerized manner to obtain a container image corresponding to the application to be collected includes: Deploy each of the multiple applications to be collected in a containerized manner to obtain container images corresponding to the multiple applications to be collected one by one; The performing a data collection operation on the container image to obtain observable data of the application to be collected includes: The data collection operation is performed on each container image corresponding to the multiple application programs to be collected, so as to obtain observable data of each of the multiple application programs to be collected.

3. The data collection method according to claim 2, characterized in that: The performing the data collection operation on each container image corresponding to the plurality of application programs to be collected respectively includes: Performing the data collection operation on each container image corresponding to the multiple application programs to be collected in sequence; After obtaining the observable data of each of the plurality of applications to be collected, the method further includes: A test report is generated based on the observable data of each of the multiple application programs to be collected.

4. The data collection method according to any one of claims 1 to 3, characterized in that: Before using the data collection program to collect data from the container image, the data collection operation further includes: The sample data is imported into the container image, where the sample data is working data of the application to be collected in a real working environment.

5. The data collection method according to any one of claims 1 to 3, characterized in that: After starting the container image, the data collection operation further includes: The startup parameters of the corresponding application to be collected are set in the container image.

6. The data collection method according to any one of claims 1 to 3, characterized in that: The step of performing data collection operation on the container image is performed according to a preset cycle, or the step of performing data collection operation on the container image is performed when the application to be collected changes, or the step of performing data collection operation on the container image is performed when the data collection program changes.

7. The data collection method according to any one of claims 1 to 3, characterized in that: After the data collection program is used to collect data from the container image, the data collection operation further includes: Perform integrity check on the collected observable data.

8. The data collection method according to any one of claims 1 to 3, characterized in that: After the data collection program is used to collect data from the container image, the data collection operation further includes: The data type of the collected observable data is compared with the expected type to determine whether the data type of the collected observable data meets expectations.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: A computer program / instruction is stored, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images