Application data management system and processing method, device and server

Abstract

The embodiment of the invention discloses an application data management system, an application data processing method, an application data processing device and a server. The system comprises an asynchronous communication component, a message publishing and subscribing component, a column database and a data consumption process group, and the asynchronous communication component caches application data asynchronously reported by different service applications; wherein the asynchronous communication component periodically writes the application data into the publish-subscribe message component, and the data consumption process group periodically pulls the application data from the publish-subscribe message component in parallel and transfers the application data into the column database. According to the technical scheme provided by the embodiment of the invention, the application data can still be stored according to the specified rhythm under high-concurrency application data reporting, the peak clipping is carried out on the storage efficiency of the application data, the high efficiency and stability of application data processing are improved, and the load pressure of a columntype database is greatly reduced; meanwhile, excessive machines do not need to be deployed in the service cluster, so that the deployment complexity of application data management is reduced.

Description

technical field

[0001] The embodiments of the present invention relate to the technical field of data processing, and in particular to a management system, processing method, device and server of application data. Background technique

[0002] At present, a large number of application programs have been developed for different business types in the Internet field. When each application program performs any business operation, it will call the corresponding interface to report the corresponding application data to the server. Class application data is stored in the corresponding database, and then a daemon process is used to monitor the abnormality of the application data in the database; however, when the server does not respond in time due to excessive data reporting, it will directly affect the user The business execution on the application program and the high-concurrency data reporting will cause the storage bottleneck of the database.

[0003] In response to the above ...

Examples

Embodiment 1

[0028] figure 1 It is a schematic architecture diagram of an application data management system provided by Embodiment 1 of the present invention. This embodiment is applicable to the situation of managing the application data reported by any kind of service application. Specifically, refer to figure 1 , the application data management system 10 may include: an asynchronous communication component 110 , a publish-subscribe message component 120 , a columnar database 130 and a data consumption process group 140 .

[0029] Among them, the asynchronous communication component 110 caches the application data reported asynchronously by different business applications; specifically, the asynchronous communication component 110 periodically writes the application data into the publish-subscribe message component 120, and the data consumption process group 140 regularly writes the application data The application data is fetched in parallel and transferred to the columnar database 1...

Embodiment 2

[0043] figure 2 It is a schematic structural diagram of an application data management system provided by Embodiment 2 of the present invention. This embodiment is optimized on the basis of the technical solutions provided by the above embodiments. refer to figure 2 , the application data management system 20 may include an asynchronous communication component 201 , a publish-subscribe message component 202 , a columnar database 203 , a data consumption process group 204 and a load balancer 205 .

[0044] Wherein, the asynchronous communication component 201 can cache application data asynchronously reported by different business applications.

[0045] Specifically, the asynchronous communication component 201 regularly writes application data into the publish-subscribe message component 202, and the data consumption process group 204 periodically pulls application data from the publish-subscribe message component 202 in parallel and transfers them to the columnar database...

Embodiment 3

[0057] image 3 It is a flow chart of a method for processing application data provided by Embodiment 3 of the present invention. This embodiment is applicable to the situation of managing application data reported by any service application. The application data processing method provided in this embodiment can be executed by the application data processing device provided in the embodiment of the present invention, and the device can be realized by means of software and / or hardware, and integrated in the server executing the method , the server may be a background server corresponding to the application data management system provided in any embodiment of the present invention.

[0058] Specifically, refer to image 3 , the method may include the following steps:

[0059] S310. Regularly write the application data asynchronously reported by different business applications into the publish-subscribe message component through the asynchronous communication component.

[006...

technical field

[0001] The embodiments of the present invention relate to the technical field of data processing, and in particular to a management system, processing method, device and server of application data. Background technique

[0002] At present, a large number of application programs have been developed for different business types in the Internet field. When each application program performs any business operation, it will call the corresponding interface to report the corresponding application data to the server. Class application data is stored in the corresponding database, and then a daemon process is used to monitor the abnormality of the application data in the database; however, when the server does not respond in time due to excessive data reporting, it will directly affect the user The business execution on the application program and the high-concurrency data reporting will cause the storage bottleneck of the database.

[0003] In response to the above ...

Examples

Embodiment 1

[0028] figure 1 It is a schematic architecture diagram of an application data management system provided by Embodiment 1 of the present invention. This embodiment is applicable to the situation of managing the application data reported by any kind of service application. Specifically, refer to figure 1 , the application data management system 10 may include: an asynchronous communication component 110 , a publish-subscribe message component 120 , a columnar database 130 and a data consumption process group 140 .

[0029] Among them, the asynchronous communication component 110 caches the application data reported asynchronously by different business applications; specifically, the asynchronous communication component 110 periodically writes the application data into the publish-subscribe message component 120, and the data consumption process group 140 regularly writes the application data The application data is fetched in parallel and transferred to the columnar database 1...

Embodiment 2

[0043] figure 2 It is a schematic structural diagram of an application data management system provided by Embodiment 2 of the present invention. This embodiment is optimized on the basis of the technical solutions provided by the above embodiments. refer to figure 2 , the application data management system 20 may include an asynchronous communication component 201 , a publish-subscribe message component 202 , a columnar database 203 , a data consumption process group 204 and a load balancer 205 .

[0044] Wherein, the asynchronous communication component 201 can cache application data asynchronously reported by different business applications.

[0045] Specifically, the asynchronous communication component 201 regularly writes application data into the publish-subscribe message component 202, and the data consumption process group 204 periodically pulls application data from the publish-subscribe message component 202 in parallel and transfers them to the columnar database...

Embodiment 3

[0057] image 3 It is a flow chart of a method for processing application data provided by Embodiment 3 of the present invention. This embodiment is applicable to the situation of managing application data reported by any service application. The application data processing method provided in this embodiment can be executed by the application data processing device provided in the embodiment of the present invention, and the device can be realized by means of software and / or hardware, and integrated in the server executing the method , the server may be a background server corresponding to the application data management system provided in any embodiment of the present invention.

[0058] Specifically, refer to image 3 , the method may include the following steps:

[0059] S310. Regularly write the application data asynchronously reported by different business applications into the publish-subscribe message component through the asynchronous communication component.

[006...

Images