Service change data processing method and device, equipment and storage medium

Abstract

The invention discloses a business change data processing method and device, equipment and a storage medium. The business change data processing method comprises the following steps: acquiring business information in real time; obtaining a plurality of first data sources in the service information; determining a classification category of each first data source; and based on the classification categories, performing parallel streaming computation on the plurality of first data sources to obtain a plurality of streaming computation results, and outputting the streaming computation results. According to the invention, based on the classification category of each first data source, the plurality of first data sources are subjected to parallel streaming computation to obtain and output the plurality of streaming computation results, so that the data sources are subjected to parallel streaming computation, and the data can be processed while being generated, so that the change state of the service information data can be sensed in real time; moreover, the data does not need to be stored in a disk or a database for data calling, so that the delay is greatly reduced, and the data processing and calculation efficiency when the business is changed can be greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a method, device, equipment and storage medium for processing business change data. Background Art

[0002] With the rapid development of the Internet and the continuous deepening of various Internet network applications, the era of big data has arrived. The amount of Internet data is growing rapidly, and the data structures and data types that need to be processed are becoming more and more diverse. At present, the demand for data in the application market is increasing day by day. In order to ensure the timeliness and quality of data services, clients usually need to call data from third-party databases.

[0003] However, with the explosive growth of data, traditional data processing models are no longer suitable for analyzing massive amounts of data. They cannot perceive the changing state of data in real time, and the data calculation efficiency is slow when the business changes. Summary of the invention

[0004] The present invention provides a method, device, equipment and storage medium for processing business change data to solve the above technical problems and effectively improve the processing and calculation efficiency of data during business changes.

[0005] In order to solve the above technical problems, the present invention provides a method for processing service change data, comprising:

[0006] Get business information in real time;

[0007] Acquire several first data sources in the business information;

[0008] Determining classification categories of each first data source;

[0009] Based on the classification categories, parallel streaming computing is performed on a plurality of first data sources to obtain and output a plurality of streaming computing results.

[0010] The beneficial effects of the present invention are:

[0011] The present invention obtains several first data sources in the business information, and based on the classification category of each first data source, performs parallel streaming calculations on the several first data sources to obtain and output several streaming calculation results. Therefore, the present invention can process the data while it is generated by performing parallel streaming calculations on the data sources to achieve real-time perception of the changing state of the business information data, and there is no need to store the data on the disk or in the database for data call, thereby greatly reducing delays and greatly improving the processing and calculation efficiency of data when the business changes.

[0012] As a preferred solution, the real-time acquisition of business information includes:

[0013] Adopt distributed deployment mode to deploy several business information collection systems;

[0014] Control several business information collection systems to collect initial business information in real time;

[0015] The initial business information is preprocessed to obtain business information.

[0016] This preferred solution controls several distributedly deployed business information collection systems to collect initial business information in real time, and can realize multi-segment, multi-node, and full-path collection of real-time business information to ensure the comprehensiveness of business information.

[0017] As a preferred solution, the preprocessing of the initial service information to obtain the service information includes:

[0018] Acquire several initial data sources in the initial business information;

[0019] Cleaning several initial data sources to perform cluster analysis on the several initial data sources to obtain classification categories of each initial data source;

[0020] The initial business information is updated based on the classification category of each initial data source to obtain business information.

[0021] This preferred solution cleans several initial data sources to cluster them, can cluster duplicate data or similar data, and obtain classification categories for each initial data source, so as to facilitate subsequent parallel streaming calculations on the data based on the classification categories, thereby improving the convenience and efficiency of subsequent calculations.

[0022] As a preferred solution, the method of performing parallel stream computing on a plurality of first data sources based on the classification categories, obtaining and outputting a plurality of stream computing results, includes:

[0023] Based on the classification categories, clustering the first data sources to obtain a plurality of cluster classifications; the cluster classifications include message queue data, database, cache and file;

[0024] Using different streaming computing frameworks, streaming computing is performed on multiple cluster classifications at the same time to obtain several streaming computing results;

[0025] Based on business scenarios and application time response requirements, multiple streaming computing results are encapsulated into multiple instances for deployment and output.

[0026] This preferred solution clusters several first data sources based on classification categories to obtain multiple cluster classifications, and then adopts different streaming computing frameworks to perform streaming computing on multiple cluster classifications at the same time to obtain several streaming computing results. It can perceive the changing status of business data in real time, greatly improving the convenience and efficiency of business data processing and computing.

[0027] As a preferred solution, after performing parallel stream computing on a plurality of first data sources based on the classification categories, obtaining and outputting a plurality of stream computing results, the method further includes:

[0028] Perform validity verification on several stream computing results to obtain several verification data;

[0029] Write several verification data into the preset database.

[0030] After the validity of the acquired streaming computing results is verified, the verification data is written into the preset database. Through the validity verification, invalid or erroneous data can be filtered out to avoid system errors or anomalies caused by erroneous or invalid data, thereby ensuring the accuracy and quality of the stored data, thereby improving the user experience of the client calling the business data.

[0031] As a preferred solution, the method for processing business change data further includes:

[0032] Obtaining a hot standby copy or a replayable data copy corresponding to each first data source;

[0033] When a data failure occurs in the first data source, the hot standby copy replaces the first data source, or the replayable data copy is enabled to update the first data source.

[0034] In this preferred solution, each first data source has a corresponding hot standby copy or a replayable data copy. When the first data source fails, the hot standby copy can be used to take over and reduce the interruption time; or a replayable data copy can be added at the same time. When the hot standby copy fails to take over, the replayable data copy can be used to restart and release the lost data, so as to further reduce the impact of the interruption, ensure the continuity of data supply to customers, and ensure customers' usage experience of business data.

[0035] Accordingly, in order to solve the above technical problems, the present invention also provides a device for processing business change data, comprising: a data acquisition module, a first acquisition module, a category determination module and a calculation processing module;

[0036] Wherein, the data acquisition module is used to acquire business information in real time;

[0037] The first acquisition module is used to acquire a plurality of first data sources in the business information;

[0038] The category determination module is used to determine the classification category of each first data source;

[0039] The computing processing module is used to perform parallel streaming computing on a plurality of first data sources based on classification categories, obtain a plurality of streaming computing results and output them.

[0040] As a preferred solution, the data acquisition module includes: a system deployment unit, a real-time acquisition unit and an information processing unit;

[0041] The system deployment unit is used to deploy several business information collection systems in a distributed deployment manner;

[0042] The real-time acquisition unit is used to control a plurality of business information acquisition systems to acquire initial business information in real time;

[0043] The information processing unit is used to pre-process the initial service information to obtain service information.

[0044] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for processing service change data as described in any one of the above items are implemented.

[0045] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for processing business change data as described in any one of the above items are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 : A flow chart of an embodiment of a method for processing business change data provided by the present invention;

[0047] Figure 2 : A structural diagram of an embodiment of a device for processing service change data provided by the present invention;

[0048] Figure 3 : A structural diagram of an embodiment of a data acquisition module provided by the present invention;

[0049] Figure 4 : A structural diagram of an embodiment of a terminal device of the hardware operating environment provided by the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] With the rapid development of the Internet and the continuous deepening of various Internet network applications, the era of big data has arrived. The amount of Internet data is growing rapidly, and the data structures and data types that need to be processed are becoming more and more diverse. At present, the demand for data in the application market is increasing day by day. In order to ensure the timeliness and quality of data services, clients usually need to call data from third-party databases.

[0052] However, with the explosive growth of data, traditional data processing models are no longer suitable for analyzing massive amounts of data. They cannot perceive the changing state of data in real time, and the data calculation efficiency is slow when the business changes.

[0053] Embodiment 1

[0054] In order to solve the above technical problems, the present invention provides a method, device, equipment and storage medium for processing business change data to solve the above technical problems and effectively improve the processing and calculation efficiency of data when the business changes.

[0055] Please refer to Figure 1 , is a flow chart of an embodiment of a method for processing service change data provided by an embodiment of the present invention. Figure 1 As shown, the method for processing the business change data includes steps 101 to 104, specifically:

[0056] Step 101: Obtain business information in real time;

[0057] Step 102: Acquire several first data sources in the business information;

[0058] Step 103: Determine the classification category of each first data source;

[0059] Step 104: Based on the classification categories, parallel streaming computing is performed on the plurality of first data sources to obtain and output a plurality of streaming computing results.

[0060] Furthermore, in step 101, the real-time acquisition of business information includes:

[0061] Adopt distributed deployment mode to deploy several business information collection systems;

[0062] Control several business information collection systems to collect initial business information in real time;

[0063] The initial business information is preprocessed to obtain business information.

[0064] In this embodiment, the business information collection system can be deployed in a distributed manner, deployed on multiple different devices, and the multi-segment, multi-node, and full-path collection of real-time business information can be achieved through the DPDK collection method. In addition, the acquired business information can be stored according to different areas and stored in different databases according to relevant business information tags.

[0065] Further, the preprocessing of the initial service information to obtain the service information includes:

[0066] Acquire several initial data sources in the initial business information;

[0067] Cleaning is performed to perform cluster analysis on several initial data sources to obtain classification categories of each initial data source;

[0068] The initial business information is updated based on the classification category of each initial data source to obtain business information.

[0069] Compared with the prior art which generally performs data cleaning by removing erroneous data, this embodiment cleans several initial data sources by means of hash algorithm, string similarity comparison, etc. Specifically, it can determine whether the data is repeated by calculating the hash value of the data, and at the same time, it can determine the repeated data according to the similarity between the strings, so as to cluster the repeated data (or similar data) in several initial data sources; and, cleaning through the data source in the business information can effectively reduce the amount of calculation, eliminate some fixed data that does not need to be calculated, and prevent invalid calculation from occurring.

[0070] In this embodiment, when executing the above solution, a judgment criterion may be preset, and different keywords may be preset according to different attributes of each data. When the data is acquired, the data may be matched according to different keywords to perform clustering and grouping.

[0071] The present invention cleans several initial data sources to cluster them, can cluster duplicate data or similar data, and obtain classification categories of each initial data source. After cluster analysis, data in different categories can be calculated in different ways. After a certain number of iterations and the system becomes stable, parallel streaming calculations can be performed directly through different calculation methods by identifying the categories of data streams, thereby improving the convenience and efficiency of subsequent calculations.

[0072] Furthermore, in step 104, based on the classification categories, parallel stream computing is performed on the plurality of first data sources to obtain and output a plurality of stream computing results, including:

[0073] Based on the classification categories, clustering the first data sources to obtain a plurality of cluster classifications; the cluster classifications include message queue data, database, cache and file;

[0074] Using different streaming computing frameworks, streaming computing is performed on multiple cluster classifications at the same time to obtain several streaming computing results;

[0075] Based on business scenarios and application time response requirements, multiple streaming computing results are encapsulated into multiple instances for deployment and output.

[0076] In this embodiment, different business information has different data source attributes, so a variety of different data source data can be detected according to different business information, and the data source integrated within the data business information can be obtained according to the attributes.

[0077] In this embodiment, streaming computing uses a streaming computing engine as a computing processing engine. Compared with batch processing, streaming computing places more emphasis on computing data flow and low latency. It actively triggers computing jobs through real-time subscription curve data, daily / monthly frozen data, and real-time reporting events, and instantly outputs computing and analysis results, such as immediate collection and correction of collected data, statistics on data collection completeness, real-time power calculation, and intelligent analysis of power outages. Real-time streaming computing embodies the timely response capability of the new generation of metering automation master station systems.

[0078] The stream computing engine supports multiple types of data sources, including message queues, databases, caches, files, etc. According to the overall system architecture design, the computing service uses the message queue as an interface to meet the access standards of the stream computing engine. By subscribing to the message queue, the changing state of the data can be perceived in real time. In conjunction with the read operation of the cache database (similar to redis and other components), all data required for real-time computing and analysis can be accessed. Among them, the output to the cache is mainly to meet the real-time access requirements of business applications; the output to the business library can provide data access channels for business applications through data services while providing persistent storage capabilities for the calculation results.

[0079] In this embodiment, the business execution logic of real-time computing is executed in the computing job link, using the job scheduling, state management, programming model, fault-tolerant mechanism and other functions provided by the stream computing engine to develop and submit real-time computing programs, process minute data, freeze data, and report events in the data access link, and feed back the calculation results to the result output link.

[0080] In this embodiment, according to the business scenarios and application time response requirements, different real-time computing and analysis logics are encapsulated into multiple instances for deployment (for example, real-time power calculation and anomaly analysis can be developed and deployed as two independent stream computing instances), and the isolation between each instance is logically maintained. This avoids mutual interference between instances and facilitates the upgrade, operation, maintenance and management of different businesses.

[0081] The present invention clusters several first data sources based on classification categories to obtain multiple cluster classifications, and then adopts different streaming computing frameworks to perform streaming computing on multiple cluster classifications at the same time to obtain several streaming computing results. The changing state of business data can be perceived in real time, greatly improving the convenience and efficiency of business data processing and computing.

[0082] Furthermore, after performing parallel stream computing on a plurality of first data sources based on the classification categories to obtain and output a plurality of stream computing results, the method further includes:

[0083] Perform validity verification on several stream computing results to obtain several verification data;

[0084] Write several verification data into the preset database.

[0085] In this embodiment, the rationality of the data can be verified (excluding the abnormality of meter code reverse running), for example: meter reading settlement data (frozen data, etc.). In addition, when the real-time requirements for power outage alarm events are high, it is required to quickly process related alarms through real-time stream computing, such as the user's power outage event analysis, and quickly determine the location of the power outage through multiple interrelated alarm events.

[0086] After the present invention performs validity verification on the obtained streaming computing results, the verification data is written into a preset database. Through validity verification, invalid or erroneous data can be filtered out to avoid system errors or anomalies caused by erroneous or invalid data, thereby ensuring the accuracy and quality of the stored data, thereby improving the user experience of the client calling business data.

[0087] Furthermore, the method for processing business change data further includes:

[0088] Obtaining a hot standby copy or a replayable data copy corresponding to each first data source;

[0089] When a data failure occurs in the first data source, the hot standby copy replaces the first data source, or the replayable data copy is enabled to update the first data source.

[0090] In the present invention, each first data source has a corresponding hot standby copy or a replayable data copy. When the first data source fails, it can be taken over by the hot standby copy to reduce the interruption time; or a replayable data copy can be added at the same time. When the hot standby copy fails to take over, the replayable data copy can be used to restart and release the lost data to further reduce the impact of the interruption, ensure the continuity of data supply to customers, and ensure customers' usage experience of business data.

[0091] Accordingly, in order to solve the above technical problems, the present invention also provides a device for processing business change data, please refer to Figure 2 , which is a structural diagram of an embodiment of a device for processing business change data provided by the present invention.

[0092] like Figure 2 As shown, the service change data processing device 20 includes: a data acquisition module 201, a first acquisition module 202, a category determination module 203 and a calculation processing module 204;

[0093] Wherein, the data acquisition module 201 is used to acquire business information in real time;

[0094] The first acquisition module 202 is used to acquire a plurality of first data sources in the business information;

[0095] The category determination module 203 is used to determine the classification category of each first data source;

[0096] The computing processing module 204 is used to perform parallel streaming computing on a plurality of first data sources based on classification categories, obtain a plurality of streaming computing results and output them.

[0097] Please refer to Figure 3 , Figure 3 A schematic structural diagram of an embodiment of a data acquisition module provided by the present invention.

[0098] Furthermore, if Figure 3 As shown, the data acquisition module 201 includes: a system deployment unit 2011, a real-time collection unit 2012 and an information processing unit 2013;

[0099] The system deployment unit 2011 is used to deploy several business information collection systems in a distributed deployment manner;

[0100] The real-time collection unit 2012 is used to control a plurality of business information collection systems to collect initial business information in real time;

[0101] The information processing unit 2013 is used to pre-process the initial service information to obtain service information.

[0102] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for processing service change data as described in any one of the above items are implemented.

[0103] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for processing business change data as described in any one of the above items are implemented.

[0104] To apply the embodiments of the present invention, please refer to Figure 4 , Figure 4 A schematic structural diagram of an embodiment of a terminal device of the hardware operating environment provided by the present invention.

[0105] In the embodiment of the present invention, the terminal device can be a PC, or it can be a smart phone, a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a portable computer, or other portable terminal devices with display functions.

[0106] like Figure 4 As shown, the terminal device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0107] Optionally, the terminal device may also include a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like. Among them, sensors include light sensors, motion sensors, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display screen according to the brightness of the ambient light, and the proximity sensor may turn off the display screen and / or backlight when the mobile terminal device is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile terminal device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; of course, the mobile terminal device may also be equipped with other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., which will not be repeated here.

[0108] Those skilled in the art will understand that Figure 4 The terminal device structure shown in the figure does not constitute a limitation on the terminal device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0109] like Figure 4 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a data calculation program.

[0110] exist Figure 4 In the terminal device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the data calculation program stored in the memory 1005.

[0111] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0112] The present invention obtains several first data sources in the business information, and based on the classification category of each first data source, performs parallel streaming calculations on the several first data sources to obtain and output several streaming calculation results. Therefore, the present invention can process the data while it is generated by performing parallel streaming calculations on the data sources to achieve real-time perception of the changing state of the business information data, and there is no need to store the data on the disk or in the database for data call, thereby greatly reducing delays and greatly improving the processing and calculation efficiency of data when the business changes.

[0113] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for processing business change data, characterized in that: include: Get business information in real time; Acquire several first data sources in the business information; Determining classification categories of each first data source; Based on the classification categories, parallel streaming computing is performed on a plurality of first data sources to obtain and output a plurality of streaming computing results.

2. A method for processing business change data according to claim 1, characterized in that: The real-time acquisition of business information includes: Adopt distributed deployment mode to deploy several business information collection systems; Control several business information collection systems to collect initial business information in real time; The initial business information is preprocessed to obtain business information.

3. A method for processing business change data according to claim 2, characterized in that: The preprocessing of the initial service information to obtain the service information includes: Acquire several initial data sources in the initial business information; Cleaning several initial data sources to perform cluster analysis on the several initial data sources to obtain classification categories of each initial data source; The initial business information is updated based on the classification category of each initial data source to obtain business information.

4. A method for processing business change data according to claim 1, characterized in that: The method of performing parallel stream computing on a plurality of first data sources based on the classification categories, obtaining and outputting a plurality of stream computing results, includes: Based on the classification categories, clustering the first data sources to obtain a plurality of cluster classifications; the cluster classifications include message queue data, database, cache and file; Using different streaming computing frameworks, streaming computing is performed on multiple cluster classifications at the same time to obtain several streaming computing results; Based on business scenarios and application time response requirements, multiple streaming computing results are encapsulated into multiple instances for deployment and output.

5. A method for processing business change data according to claim 1, characterized in that: After performing parallel stream computing on a plurality of first data sources based on the classification categories to obtain and output a plurality of stream computing results, the method further includes: Perform validity verification on several stream computing results to obtain several verification data; Write several verification data into the preset database.

6. A method for processing business change data according to claim 1, characterized in that: Also includes: Obtaining a hot standby copy or a replayable data copy corresponding to each first data source; When a data failure occurs in the first data source, the hot standby copy replaces the first data source, or the replayable data copy is enabled to update the first data source.

7. A device for processing business change data, characterized in that: include: A data acquisition module, a first acquisition module, a category determination module and a calculation processing module; Wherein, the data acquisition module is used to acquire business information in real time; The first acquisition module is used to acquire a plurality of first data sources in the business information; The category determination module is used to determine the classification category of each first data source; The computing processing module is used to perform parallel streaming computing on a plurality of first data sources based on classification categories, obtain a plurality of streaming computing results and output them.

8. A device for processing service change data according to claim 7, characterized in that: The data acquisition module includes: a system deployment unit, a real-time collection unit and an information processing unit; The system deployment unit is used to deploy several business information collection systems in a distributed deployment manner; The real-time acquisition unit is used to control a plurality of business information acquisition systems to acquire initial business information in real time; The information processing unit is used to pre-process the initial service information to obtain service information.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for processing business change data as claimed in any one of claims 1 to 6 are implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for processing service change data as claimed in any one of claims 1 to 6 are implemented.

Images