Data flow loosely coupled statistical method, device, equipment, medium and program product

Abstract

The embodiment of the disclosure provides a data stream loose coupling statistical method, which relates to the field of financial technology and big data, and the method comprises the following steps: acquiring a business data stream, wherein the business data stream is the sum of real-time data of multiple types of businesses; extracting basic data and real-time business data of each type of business from the business data stream, wherein the basic data comprises associated data between businesses having a dependency relationship; supplementing the basic data to the real-time business data of each type of business to obtain loose coupling business data of each type of business; and storing the loose coupling business data of each type of business in a distributed message queue respectively to perform classified statistics. The disclosure also provides a data stream loose coupling statistical device, equipment, storage medium and program product.

Description

Technical Field

[0001] This disclosure relates to the field of big data, and more specifically to a loosely coupled statistical method, apparatus, device, medium, and program product for data streams. Background Technology

[0002] In the financial industry, transaction data generated during business operations is recorded in the business database in real time. Later, data warehouse technology (Extract-Transform-Load, or ETL) is generally used to import the data into the analysis database when the business system stops service, and the data is summarized, providing strong data support for the daily operations of branches.

[0003] With the development of the financial industry, many application scenarios require real-time statistical data to gain valuable time for the implementation of financial strategies. Due to the complexity of business logic in the financial industry, the degree and complexity of interdependence among various business data gradually increase as business logic expands, meaning the coupling between data becomes higher. This high degree of data coupling leads to increasing difficulty in achieving system and business scalability, which to some extent limits system scalability. Therefore, it is necessary to research a loosely coupled business partitioning solution to address this complex situation. Summary of the Invention

[0004] In view of the above problems, this disclosure provides data flow loose coupling statistics methods, apparatus, devices, media and program products to improve the degree of data loose coupling.

[0005] According to a first aspect of this disclosure, a loosely coupled data stream statistics method is provided, the method comprising: acquiring a business data stream, the business data stream being the sum of real-time data of multiple types of businesses; extracting basic data and real-time business data of each type of business from the business data stream, the basic data including correlation data between businesses with dependencies; supplementing the basic data into the real-time business data of each type of business to obtain loosely coupled business data of each type of business; and storing the loosely coupled business data of each type of business into a distributed message queue for classification and statistics.

[0006] According to embodiments of this disclosure, the extraction of basic data and real-time business data of various types of services from the business data stream includes: copying the business data stream into a first business data stream and a second business data stream based on a preset copy command; filtering the first business data stream based on preset basic keywords to select basic data corresponding to the basic keywords from the business data stream; and classifying the second business data stream according to the business type to obtain real-time business data of each type of service.

[0007] According to an embodiment of this disclosure, the step of filtering the first business data stream based on preset basic keywords to obtain the basic data further includes: constructing basic data key-value pairs with the basic keywords and the basic data values, and storing them in the basic data retrieval library of the Redis data structure.

[0008] According to embodiments of this disclosure, supplementing the basic data into the real-time business data of various types of services to obtain loosely coupled business data for each type of service includes: obtaining basic keywords of the basic data that each type of service depends on based on the dependency relationship between services; calling the basic data retrieval library to obtain the basic data corresponding to the basic keywords; and supplementing the basic data into the real-time business data of services that depend on the basic data to obtain the loosely coupled business data.

[0009] According to embodiments of this disclosure, the step of storing loosely coupled business data of various types of services into a distributed message queue for classification and statistics includes: using a distributed near real-time computing engine to read loosely coupled business data of various types of services from the distributed message queue, and statistically analyzing the loosely coupled business data of each type of service; summarizing the statistically analyzed loosely coupled business data of each type of service to obtain the loosely coupled statistical data of the business data stream.

[0010] According to embodiments of this disclosure, the method further includes: setting up a Redis server cluster, the Redis server cluster including one master Redis server group and multiple slave Redis server groups, the master Redis server group being used to execute the loosely coupled data flow statistics method, and the slave Redis server groups being used to back up data generated on the master Redis server; monitoring the operating status of the Redis server cluster; and when the master Redis server group fails, upgrading one of the multiple slave Redis server groups to a Redis server.

[0011] A second aspect of this disclosure provides a loosely coupled data stream statistics device, comprising: a data stream acquisition module for acquiring a business data stream, wherein the business data stream is the sum of real-time data of multiple types of businesses; a data extraction module for extracting basic data and real-time business data of each type of business from the business data stream, wherein the basic data includes correlation data between businesses with dependencies; a data completion module for supplementing the basic data into the real-time business data of each type of business to obtain loosely coupled business data of each type of business; and a distributed statistics module for storing the loosely coupled business data of each type of business into a distributed message queue for performing classification statistics.

[0012] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the aforementioned loosely coupled data flow statistics method.

[0013] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the aforementioned loosely coupled statistical method for data flow.

[0014] The fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described loosely coupled statistical method for data flow.

[0015] The above-described at least one technical solution adopted in the embodiments of this disclosure can achieve the following beneficial effects:

[0016] According to the loosely coupled statistical method for data flow provided in this disclosure, while acquiring the data flow, the basic data on which each business requirement depends is obtained in real time, so as to complete the business data of each type of business, remove the data dependency relationship between each business, thereby realizing loose coupling of data between various types of businesses, facilitating the independence of subsequent business logic and computational isolation, and thus enabling rapid business expansion and convenient business deployment. Attached Figure Description

[0017] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0018] Figure 1 The illustration schematically depicts an application scenario of a loosely coupled statistical method, apparatus, device, medium, and program product for data flow according to embodiments of the present disclosure.

[0019] Figure 2 A flowchart illustrating a loosely coupled statistical method for data flow according to an embodiment of the present disclosure is shown schematically.

[0020] Figure 3 A detailed flowchart of a loosely coupled statistical method for data flow according to an embodiment of the present disclosure is illustrated schematically;

[0021] Figure 4 This illustration schematically shows an application diagram of the loosely coupled statistical method for data flow according to embodiments of the present disclosure;

[0022] Figure 5 A schematic diagram of a Redis server cluster according to an embodiment of the present disclosure is shown.

[0023] Figure 6A schematic block diagram of a loosely coupled statistical device for data flow according to embodiments of the present disclosure is shown; and

[0024] Figure 7 A block diagram of an electronic device suitable for implementing a loosely coupled statistical method for data flow, according to an embodiment of the present disclosure, is shown schematically. Detailed Implementation

[0025] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0027] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0028] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0029] It should be noted that the loosely coupled statistical method and apparatus for data flow disclosed herein can be used for data statistics in the financial field, as well as for data statistics in any field other than finance. The application fields of the loosely coupled statistical method and apparatus for data flow disclosed herein are not limited.

[0030] The embodiments of this disclosure provide a loosely coupled statistical method for data streams, which involves acquiring a business data stream, where the business data stream is the sum of real-time data from multiple types of businesses; extracting basic data and real-time business data for each type of business from the business data stream, where the basic data includes relational data between businesses with dependencies; supplementing the real-time business data of each type of business with the basic data to obtain loosely coupled business data for each type of business; and storing the loosely coupled business data for each type of business in a distributed message queue for classification and statistical analysis.

[0031] Figure 1 The illustration schematically depicts an application scenario of a loosely coupled statistical method, apparatus, device, medium, and program product for data flow according to embodiments of the present disclosure.

[0032] like Figure 1 As shown, application scenario 100 according to this embodiment may include data statistics in the financial field. Network 104 is used as a medium to provide a communication link between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired, wireless communication links or fiber optic cables, etc.

[0033] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0034] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0035] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0036] It should be noted that the loosely coupled data flow statistics method provided in this embodiment can generally be executed by server 105. Correspondingly, the loosely coupled data flow statistics device provided in this embodiment can generally be located in server 105. The loosely coupled data flow statistics method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the loosely coupled data flow statistics device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0037] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0038] The following will be based on Figure 1 The described scene, through Figures 2-3 The loosely coupled statistical method for data flow according to the disclosed embodiments is described in detail.

[0039] Figure 2 A flowchart illustrating a loosely coupled statistical method for data flow according to an embodiment of this disclosure is shown schematically.

[0040] like Figure 2 As shown, the loosely coupled statistical method for data flow in this embodiment includes operations S210 to S240, and the transaction processing method can be executed sequentially.

[0041] In operation S210, the business data stream is acquired, which is the sum of real-time data from various types of business.

[0042] In this embodiment, the business data stream can be the enterprise's business ledger data, which includes the data flow of various business operations within the enterprise.

[0043] Because various business processes may have dependencies, for example, in a bank's loan business, loan approval depends on the borrower's credit data for judgment and review. Only when the review is approved can the loan process proceed smoothly. This means that processing a single business may require the coordinated calculation of data from multiple business processes, increasing the computational complexity of the server. According to operations S220-S240 of the embodiments of this disclosure, loose coupling of data between different business processes can be achieved.

[0044] In operation S220, basic data and real-time business data of various types of business are extracted from the business data stream. The basic data includes the correlation data between businesses that have dependencies.

[0045] In this embodiment of the disclosure, the data that each business depends on is filtered out as basic data from the business data stream. For example, in financial business, users need to use their account information in consumption businesses such as car purchase, home renovation, and installment payment. That is, multiple consumption businesses depend on the user's account opening business information. Therefore, the account information in the user's account opening business is the basic information that the above-mentioned consumption businesses need to rely on.

[0046] In this embodiment of the disclosure, different business data are first separated from the business data stream according to business type, in order to prepare for the subsequent implementation of loose coupling of data for different types of business.

[0047] In operation S230, basic data is supplemented into the real-time business data of each type of business to obtain loosely coupled business data for each type of business.

[0048] In this embodiment of the disclosure, the basic data on which each type of service depends is supplemented into the data of each service, thereby decoupling the dependencies between different services and no longer relying on other services to obtain basic data, thus achieving loose coupling of data for a single service.

[0049] In operation S240, loosely coupled business data of various types of services are stored in distributed message queues for classification and statistics.

[0050] According to the loosely coupled statistical method for data flow provided in this disclosure, business data of various types of services can be completed, and additional basic data on which their business needs depend can be obtained in real time. This removes the data dependency between various services, realizes loose coupling between various types of services, facilitates the independence of subsequent business logic and computational isolation, and enables rapid business expansion and convenient business deployment.

[0051] The loosely coupled statistical method for data according to the embodiments of this disclosure will now be described in detail.

[0052] Figure 3 A detailed flowchart of a loosely coupled statistical method for data flow according to an embodiment of this disclosure is illustrated schematically.

[0053] like Figure 3 As shown, the loosely coupled statistical method for data flow in this embodiment of the present disclosure may specifically include operations S310 to S340.

[0054] In operating S310, the business data stream is acquired, which is the sum of real-time data from various types of business.

[0055] In operation S320, basic data and real-time business data of various types of business are extracted from the business data stream. The basic data includes the correlation data between businesses with dependencies. Specific operations S320 include S321 to S323.

[0056] In operation S321, the business data stream is copied into a first business data stream and a second business data stream based on a preset copy command.

[0057] In this embodiment, a replicator program can be pre-configured to execute the replication command, which copies the data in the business data stream into two copies in real time. One copy is used to extract basic keywords, and the other copy is used to classify the business data. In this way, the basic keywords and classified business data can be obtained simultaneously without affecting the original business data.

[0058] In operation S322, the first business data stream is filtered based on preset basic keywords, and the basic data corresponding to the basic keywords is selected from the business data stream.

[0059] In this embodiment, a real-time basic data filter is used to filter the first business data stream, and the basic data is written to the key-value real-time basic data retrieval database in real time. Basic data key-value pairs are constructed using basic keywords and basic data values ​​and stored in the Redis data structure's basic data retrieval database as soon as possible. This database is updated in real time to ensure the data is up-to-date. Storing basic data in key-value format in the Redis data structure's basic data retrieval database essentially means storing the key-value pairs in memory and periodically cleaning up invalid data to ensure that core basic data can be retrieved quickly.

[0060] For example, when the current data in the first business data stream is card opening information, the information includes card number, referrer, etc. Since this information will be used by multiple businesses such as payment and loan, the key-value mapping of card number and referrer will be saved to the basic data retrieval database in real time.

[0061] In operation S323, the second business data stream is classified according to the business type to obtain real-time business data for each type of business.

[0062] In this implementation, the second business data stream can be classified according to preset business category tags or keywords to obtain business data of different business types.

[0063] In operation S330, basic data is supplemented into the real-time business data of each type of service to obtain loosely coupled business data for each type of service. Specifically, operation S330 includes operations S331 to S333.

[0064] In operation S331, the basic keywords of the basic data that each type of business depends on are obtained based on the dependencies between businesses.

[0065] In this embodiment, the dependencies between various services can be clarified in the preparatory work for implementing the loosely coupled statistical method of data flow provided in this disclosure, and the source and basic keywords of the basic data on which each service depends can be sorted out so that they can be used directly in the subsequent real-time data completion process.

[0066] In operation S332, the basic data retrieval library is invoked to retrieve the basic data corresponding to the basic keywords. Using the key-value real-time basic data retrieval library allows for efficient access to massive amounts of basic data, satisfying real-time data retrieval needs and enabling real-time completion of business data.

[0067] In operation S333, basic data is added to the real-time business data of the business that depends on the basic data to obtain loosely coupled business data.

[0068] Based on the above steps, efficient retrieval of the basic data retrieval database can complete business data in real time, thereby achieving loose data coupling between business processes.

[0069] In the S340 operation, loosely coupled business data of various types of services are stored in distributed message queues for classification and statistics.

[0070] Operation S340 includes operations S341 to S342.

[0071] In operation S341, a distributed near real-time computing engine is used to read loosely coupled business data of various types of services from a distributed message queue, and to statistically analyze the loosely coupled business data of each type of service.

[0072] In operation S342, the loosely coupled business data of each type of business after statistics are summarized to obtain loosely coupled statistical data of business data flow.

[0073] Based on the above method, since the basic data is updated in real time, the business data is completed in real time, and the classification and distribution of business data are also performed in real time, the necessary conditions are provided for achieving end-to-end real-time data processing. By storing loosely coupled business data of different types of business in different message queues, subsequent statistical calculations can be easily performed using a distributed near-real-time computing engine, or independent computing engines can be deployed on different servers, achieving continuous and scalable deployment of the business.

[0074] In subsequent processing or data analysis stages, since each piece of data contains the necessary elements for statistical analysis, real-time cumulative statistics can be achieved.

[0075] Figure 4 The illustration shows a schematic diagram of the application of the loosely coupled statistical method for data flow according to an embodiment of the present disclosure.

[0076] like Figure 4 As shown, the real-time ledger of bank cards includes data flow records for various transactions, such as car purchases, installment payments, credit card usage, and credit card installment payments for home renovations, as well as data flow records for credit card issuance. These transactions are interdependent; for example, credit card usage depends on the basic data for card issuance. Given these dependencies, it presents a contradiction to also require independent partitioning and loosely coupled computation to enable independent deployment of these transactions.

[0077] According to the loosely coupled statistical method for data flow provided in this disclosure, after acquiring the business data flow, it can be used as data for pre-application processing, performing actions such as... Figures 2-3 The loosely coupled statistical method shown pre-implements loose coupling between business processes to obtain business data for each category. For example... Figure 4 As shown, after pre-processing, data for different business types, such as car purchase installment, home renovation installment, and installment payment solutions, can be extracted from the real-time ledger data. Subsequently, data processing can be performed separately for each business type to complete the transaction and further obtain statistical data for each type of data.

[0078] In the data processing of front-end applications, when a new data message stream arrives at the business data completion node, a completion program obtains basic information in real time. For example, when the arriving message is a credit card transaction message containing brief fields such as card number, transaction amount, and time, the completion node program retrieves the cardholder's basic information (such as ID number) from the key-value real-time basic information database based on the card number in the message. It can also retrieve the marketing personnel's information in real time based on the card number. Therefore, when this brief information is completed through necessary business data, forming a set of fields with more information in real time, this information will be synchronously distributed to downstream business topics (such as car purchase installment topics, home renovation installment topics, cash topics, etc.) according to the message type, thus achieving real-time message completion and distribution.

[0079] In this embodiment, the design of the multidimensional key-value real-time basic data retrieval library must meet at least the following requirements: First, it must be able to efficiently access massive amounts of basic data. For example, in banking, it mainly stores basic information such as credit card issuance and authorization. Due to the large number of users in the banking system, the base amount of this basic information is quite large. Second, it must also be able to perform real-time data analysis of various business types, such as consumption, immediately after card issuance. Since data analysis is real-time, its prerequisite is that the multidimensional key-value real-time basic data retrieval library must be real-time. Third, it must meet the high availability requirements of core business operations. Therefore, in this embodiment, a Redis server cluster is used to ensure the stable implementation of the loosely coupled statistical method for data flow provided in this embodiment.

[0080] It should be noted that the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information involved in the technical solution disclosed herein all comply with the provisions of relevant laws and regulations, have taken necessary confidentiality measures, and do not violate public order and good morals.

[0081] Figure 5 The illustration shows an application diagram of a Redis server cluster according to an embodiment of the present disclosure.

[0082] like Figure 5 As shown in the embodiments of this disclosure, the loosely coupled data flow statistics method can be implemented using a Redis server cluster. The implementation method includes: setting up a Redis server cluster, which includes one master Redis server group and multiple slave Redis server groups. The master Redis server group is used to execute the loosely coupled data flow statistics method, and the slave Redis server groups are used to back up data generated on the master Redis server in real time. The data includes business data flow, basic data and business data extracted from the business data flow, and message queues storing slave coupled data of various businesses; setting up a sentinel program to monitor the running status of the Redis server cluster; when the master Redis server group fails, the sentinel program upgrades one of the multiple slave Redis server groups to a Redis server, and then executes the loosely coupled data flow statistics method provided in the embodiments of this disclosure, thereby ensuring that the real-time processing of data is not affected.

[0083] Based on the above-described loosely coupled data flow statistics method, this disclosure also provides a loosely coupled data flow statistics device. The following will be combined with... Figure 5 The device is described in detail.

[0084] Figure 6 A schematic block diagram of a loosely coupled statistical device for data flow according to an embodiment of the present disclosure is shown.

[0085] like Figure 6 As shown, the loosely coupled data flow statistics device 600 of this embodiment includes a data flow acquisition module 610, a data extraction module 620, a data completion module 630, and a distributed statistics module 640.

[0086] The data stream acquisition module 610 is used to acquire business data streams, which are the sum of real-time data from various types of businesses. In one embodiment, the data stream acquisition module 610 can be used to perform the operation S210 described above, which will not be repeated here.

[0087] The data extraction module 620 is used to extract basic data and real-time business data of various types of businesses from the business data stream. The basic data includes correlation data between businesses that have dependencies. In one embodiment, the data extraction module 620 can be used to perform the operation S220 described above, which will not be repeated here.

[0088] The data completion module 630 is used to supplement the basic data into the real-time business data of various types of services, thereby obtaining loosely coupled business data for each type of service. In one embodiment, the data completion module 630 can be used to perform the operation S230 described above, which will not be repeated here.

[0089] The distributed statistics module 640 is used to store loosely coupled business data of various types of services into distributed message queues for classification and statistics. In one embodiment, the distributed statistics module 640 can be used to execute the operation S240 described above, which will not be repeated here.

[0090] According to embodiments of this disclosure, any multiple modules among the data stream acquisition module 610, data extraction module 620, data completion module 630, and distributed statistics module 640 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the data stream acquisition module 610, data extraction module 620, data completion module 630, and distributed statistics module 640 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the data stream acquisition module 610, data extraction module 620, data completion module 630, and distributed statistics module 640 can be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0091] Figure 7 A block diagram of an electronic device suitable for implementing a loosely coupled statistical method for data flow, according to an embodiment of the present disclosure, is shown schematically.

[0092] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0093] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0094] According to embodiments of this disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0095] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0096] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0097] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the loosely coupled statistical method for data flow provided in embodiments of this disclosure.

[0098] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0099] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0100] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0101] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0103] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0104] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method of loosely coupled statistics for data streams, the method comprising: The method includes: Acquire business data streams, which are the sum of real-time data from multiple types of businesses; Extract basic data and real-time business data of various types of business from the business data stream. The basic data includes the correlation data between businesses that have dependencies. The basic data is supplemented into the real-time business data of each type of business to obtain loosely coupled business data for each type of business. Loosely coupled business data for each type of business is stored in a distributed message queue for classification and statistics. The extraction of basic data and real-time business data of various types of services from the business data stream includes: The business data stream is copied into a first business data stream and a second business data stream based on a preset copy command; The first business data stream is filtered based on preset basic keywords. Basic data corresponding to the basic keywords is selected from the business data stream. Basic data key-value pairs are constructed using the basic keywords and basic data values ​​and stored in the real-time basic data retrieval library of Redis data structure. The second business data stream is classified according to the business type to obtain real-time business data for each type of business; The step of supplementing the basic data into the real-time business data of each type of business to obtain loosely coupled business data for each type of business includes: Based on the dependencies between business operations, obtain the basic keywords of the basic data that each type of business depends on; Call the real-time basic data retrieval library to obtain the basic data corresponding to the basic keywords; The basic data is supplemented into the real-time business data of the services that depend on the basic data to obtain the loosely coupled business data.

2. The method of claim 1, wherein, The step of storing loosely coupled business data of various types of services into distributed message queues for classification and statistics includes: A distributed near real-time computing engine is used to read loosely coupled business data of various types of services from a distributed message queue, and to statistically analyze the loosely coupled business data of each type of service. By summarizing the loosely coupled business data of each type of business after statistics, the loosely coupled statistical data of the business data flow is obtained.

3. The method of claim 1, wherein, The method further includes: Configure a Redis server cluster, which includes one master Redis server group and multiple slave Redis server groups. The master Redis server group is used to execute the loosely coupled data flow statistics method, and the slave Redis server groups are used to back up the data generated on the master Redis server in real time. Monitor the running status of the Redis server cluster; When the primary Redis server group fails, one of the multiple secondary Redis server groups will be promoted to a Redis server.

4. A loosely coupled data stream statistics device, said device being used to implement the method according to any one of claims 1 to 3, comprising: The data stream acquisition module is used to acquire business data streams, which are the sum of real-time data from multiple types of businesses. The data extraction module is used to extract basic data and real-time business data of various types of businesses from the business data stream. The basic data includes the correlation data between businesses that have dependencies. The data completion module is used to supplement the basic data into the real-time business data of each type of business to obtain loosely coupled business data of each type of business. The distributed statistics module is used to store loosely coupled business data of various types of business into distributed message queues for classification and statistics. The extraction of basic data and real-time business data of various types of services from the business data stream includes: The business data stream is copied into a first business data stream and a second business data stream based on a preset copy command; The first business data stream is filtered based on preset basic keywords, and basic data corresponding to the basic keywords is selected from the business data stream. The second business data stream is classified according to the business type to obtain real-time business data for each type of business.

5. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 3.

7. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 3.

Images