A big data information collection and processing system and operation method

Abstract

The present invention proposes a big data information collection and processing system and a corresponding method, which realizes information collection and management based on big data by performing data supervision and collection on the transaction behaviors of transaction nodes at the front line of financial transactions. Among them, the information data is collected by a three-part discrete collection method, and the transaction data is divided into the first transaction attribute information, the second transaction attribute information and the third transaction attribute information, which are mutually coordinated and have certain independent collection characteristics. The corresponding collection servers are respectively used for big data collection. At the same time, the collection information replacement based on risk avoidance and attack prevention is performed on a specific collection server, and the system big data security is considered in the process of executing big data transmission and system storage. Through the data normalization of the data collection and normalization module of the collection server, the big data is conditionally executed into the warehouse, thereby realizing the safe collection of big data and differentiated storage management.

Description

Technical Field

[0001] The present invention belongs to the field of information technology, and in particular relates to a big data information collection and processing system and an operation method thereof. Background Art

[0002] The requirements of information construction for information collection are becoming more stringent as the degree of informatization increases. In the process of information collection, the collection and application of big data are often mixed.

[0003] In the field, network information collection is the process of extracting unstructured information from a large number of web pages and saving it in a structured database. The information collection system is built on the basis of the network information mining engine. It can help you collect the latest information from different Internet sites in the shortest time, and publish the information to your own site in a timely manner after classification and unified format. This improves the timeliness of information and saves or reduces workload.

[0004] As a general rule for the collection of network big data information, information collection has the following seven principles, which are the most basic requirements for ensuring the quality of information collection: (1) Reliability principle: The reliability principle of information collection means that the information collected must be generated by real objects or environments, the information source must be reliable, and the collected information must reflect the real situation. The reliability principle is the basis of information collection. (2) Integrity principle: The integrity of information collection means that the collected information must be complete in content, and information collection must be in accordance with certain standards and requirements, and collect information that reflects the whole picture of things. The integrity principle is the basis of information utilization. (3) Real-time principle: The real-time nature of information collection refers to the ability to obtain the required information in a timely manner. Generally, it has three meanings: First, it refers to the time interval from the occurrence of information to the collection of information. The shorter the interval, the more timely it is. The fastest is when information collection is synchronized with information generation; second, it refers to the ability to quickly collect information when an enterprise or organization urgently needs a certain information to perform a certain task, which is called timely; third, it refers to the time spent on collecting all the information required for a certain task. The less time it takes, the faster it is. The real-time principle ensures the timeliness of information collection. (4) Principle of accuracy: The principle of accuracy means that the information collected is highly relevant to the application objectives and work requirements, the expression of the collected information is correct, it is within the scope of the collection purpose, and it is applicable and valuable to the enterprise or organization itself. The higher the degree of relevance and the stronger the adaptability, the more accurate it is. The principle of accuracy guarantees the value of information collection. (5) Principle of ease of use: The principle of ease of use means that the collected information is easy to use in a certain form. (6) Principle of planning: The information collected must not only meet current needs but also take into account future development; it must not only expand the sources of information but also persist in accumulating information over time; it should not be random, but should formulate a relatively detailed collection plan and rules and regulations based on the unit's tasks, funds, etc. (7) Principle of foresight: Information collectors must grasp the development trends of society, economy and science and technology. The information collected must not only focus on current needs but also be ahead of the times. They must be good at catching signs and trends. They must keep abreast of the future and collect predictive information that will guide future development.

[0005] As the construction of informatization deepens, the concept of big data is increasingly appearing in the practice of information collection and storage. Most scholars believe that the concept of "big data" first appeared publicly in 1998. With the rapid growth of data volume, four problems will inevitably arise, such as data being difficult to understand, difficult to obtain, difficult to process and difficult to organize. "Big Data" is used to describe this challenge, which has triggered thinking in the computing field. In 2007, Jim Gray, a pioneer in the database field, pointed out that big data will become an effective way for humans to touch, understand and approach real complex systems. He believed that after the three scientific research paradigms of experimental observation, theoretical derivation and computational simulation, the fourth paradigm - "data exploration" will be ushered in. Later, fellow scholars summarized it as "data-intensive scientific discovery", which opened up a wave of examining big data from a scientific research perspective. In 2012, Viktor Mayer-Schnberger, a professor at Oxford University, pointed out in his best-selling book Big Data: A Revolution That Will Transform How We Live, Work, and Think that data analysis will evolve from the traditional model of "random sampling", "precise solution" and "emphasis on causality" to the new model of "all data", "approximate solution" and "only look at associations without asking about causality" in the big data era, thus triggering extensive thinking and discussion on big data methods in the field of commercial applications. Big data reached its peak of publicity in 2012 and 2013. After 2014, the concept system gradually took shape, and its cognition also tended to be rational. Big data-related technologies, products, applications and standards continue to develop, gradually forming a big data ecosystem consisting of data resources and APIs, open source platforms and tools, data infrastructure, data analysis, data applications and other sectors, and continue to develop and improve. Its development hotspots show a gradual migration from technology to application and then to governance. After years of development and precipitation, people have reached a basic consensus on big data: the big data phenomenon originates from the ubiquitous application of information technology brought about by the Internet and its extensions, as well as the continuous low cost of information technology. Big data refers to a huge amount of data that cannot be acquired, managed, and processed by traditional information technology and software and hardware tools within a tolerable time. It has the characteristics of massiveness, diversity, timeliness, and variability, and requires a scalable computing architecture to support its storage, processing, and analysis. The value of big data is essentially reflected in: it provides a new way of thinking and a new means for humans to understand complex systems. In theory, at a sufficiently small time and space scale, the digitization of the real world can construct a digital virtual image of the real world, which carries the operating laws of the real world. Under the premise of having sufficient computing power and efficient data analysis methods, in-depth analysis of this digital virtual image will make it possible to understand and discover the operating behavior, status, and laws of real complex systems.It should be said that big data provides humans with a completely new way of thinking and new means to explore objective laws and transform nature and society. This is also the most fundamental reason why big data has triggered economic and social changes.

[0006] The present invention proposes a big data information collection and processing system and a corresponding method, which realizes information collection and management based on big data by performing data supervision and collection on the transaction behaviors of transaction nodes at the front line of financial transactions. Among them, the information data is collected by a three-part discrete collection method, and the transaction data is divided into the first transaction attribute information, the second transaction attribute information and the third transaction attribute information, which are mutually coordinated and have certain independent collection characteristics. The corresponding collection servers are respectively used for big data collection. At the same time, the collection information replacement based on risk avoidance and attack prevention is performed on a specific collection server, and the system big data security is considered in the process of executing big data transmission and system storage. Through the data normalization of the data collection and normalization module of the collection server, the big data is conditionally executed into the warehouse, thereby realizing the safe collection of big data and differentiated storage management. Summary of the invention

[0007] The present invention aims to provide a big data information collection and processing system and method that are superior to the prior art.

[0008] In order to achieve the above object, the technical solution of the present invention is as follows:

[0009] A big data information collection and processing system, the system comprising:

[0010] A plurality of transaction units, each of which is distributed at various locations of the financial information transaction system, performs financial information transactions, and generates transaction attribute information;

[0011] The transaction attribute information at least includes:

[0012] The first party to the transaction, the second party to the transaction, the transaction time, the transaction unit ID, the transaction settlement amount data, and the transaction verification code;

[0013] The transaction verification code is used to verify the transaction attribute information;

[0014] A first collection server, configured to collect only first transaction attribute information from each of the plurality of transaction units, and generate a first normalized data packet based on the first transaction attribute information;

[0015] The first transaction attribute information at least includes: the first transaction party and the transaction time;

[0016] A second collection server, configured to collect only second transaction attribute information from each of the plurality of transaction units, and generate a second normalized data packet based on the second transaction attribute information;

[0017] The second transaction attribute information at least includes: the second transaction party and the transaction unit ID;

[0018] a third collection server, configured to collect only third transaction attribute information from each of the plurality of transaction units, and generate a third normalized data packet based on the third transaction attribute information;

[0019] Wherein, the third transaction attribute information at least includes: transaction settlement amount data and transaction verification code;

[0020] The big data information collection and processing system also includes:

[0021] a first acquisition information replacement server, configured to replace all of the first transaction attribute information in the first normalized data packet, or replace part of the first transaction attribute information in the first normalized data packet, based on a system preset first acquisition information replacement table; and return the replaced first normalized data packet to the first acquisition server as an updated first normalized data packet of the original first normalized data packet;

[0022] An information data packet normalization module, the information data packet normalization module is used to combine the updated first normalized data packet, the second normalized data packet and the third normalized data packet into a normalized data packet;

[0023] The normalized data packet contains complete transaction attribute information of a single transaction of a single transaction unit;

[0024] The information data packet normalization module is also used to send the normalized data packet to a big data warehouse;

[0025] A big data warehouse, the big data warehouse is used to receive a normalized data packet, and verify the transaction attribute information carried by the normalized data packet based on the transaction verification code, and determine the verification result;

[0026] The big data warehouse is used to receive the normalized data packet, and verify the transaction attribute information carried by the normalized data packet based on the transaction verification code, and determine the verification result, including at least:

[0027] The transaction verification code and the corresponding transaction attribute information pre-received by the big data warehouse and notified to the big data warehouse by the verification code generation unit are compared with the transaction verification code carried by the corresponding transaction attribute information to determine whether they are the same, and if they are the same, the verification is passed;

[0028] When the verification is passed, the big data warehouse allows the transaction attribute information in the normalized data packet to be stored;

[0029] When the verification fails, the big data warehouse extracts the transaction attribute information in the normalized data packet, and based on the transaction unit ID therein, enables the system to shield the collection of transaction attribute information of the transaction unit corresponding to the transaction unit ID within a first time period.

[0030] Preferably, it also includes a verification code generating unit.

[0031] Preferably, the system presets the first collected information replacement table including the IDs of all first transaction parties and their corresponding replacement IDs, as well as the transaction time replacement strategy;

[0032] The transaction time replacement strategy is used to indicate how to replace the transaction time.

[0033] Preferably, the transaction verification code can be generated by a verification code generation unit of the system for each transaction attribute information, and the generated transaction verification code is added to the transaction attribute information and stored in the third transaction attribute information of the transaction attribute information.

[0034] At the same time, the verification code generation unit informs the big data warehouse of the transaction verification code and the corresponding transaction attribute information.

[0035] Preferably, the information data packet normalization module is hosted on a dedicated server.

[0036] At the same time, the present invention discloses an operation method of a big data information collection and processing system, the method comprising the following steps:

[0037] Step 1: operate each of the multiple transaction units, distributed at various locations of the financial information transaction system, execute financial information transactions, and generate transaction attribute information;

[0038] The transaction attribute information at least includes:

[0039] The first party to the transaction, the second party to the transaction, the transaction time, the transaction unit ID, the transaction settlement amount data, and the transaction verification code;

[0040] The transaction verification code is used to verify the transaction attribute information;

[0041] Step 2: operating the first collection server to collect only the first transaction attribute information from each of the multiple transaction units, and generating a first normalized data packet based on the first transaction attribute information;

[0042] The first transaction attribute information at least includes: the first transaction party and the transaction time;

[0043] Step 3: operating the second collection server to collect only the second transaction attribute information from each of the multiple transaction units, and generating a second normalized data packet based on the second transaction attribute information;

[0044] The second transaction attribute information at least includes: the second transaction party and the transaction unit ID;

[0045] Step 4: operating the third collection server to collect only the third transaction attribute information from each of the multiple transaction units, and generating a third normalized data packet based on the third transaction attribute information;

[0046] Wherein, the third transaction attribute information at least includes: transaction settlement amount data and transaction verification code;

[0047] Step 5: operate the first acquisition information replacement server to replace all of the first transaction attribute information in the first normalized data packet, or replace part of the first transaction attribute information in the first normalized data packet based on the system preset first acquisition information replacement table; and return the replaced first normalized data packet to the first acquisition server as an updated first normalized data packet of the original first normalized data packet;

[0048] Step 6: operate the information data packet normalization module to combine the updated first normalized data packet, the second normalized data packet and the third normalized data packet into a normalized data packet;

[0049] The normalized data packet contains complete transaction attribute information of a single transaction of a single transaction unit;

[0050] The information data packet normalization module is also used to send the normalized data packet to a big data warehouse;

[0051] Step 7: operate the big data warehouse to receive the normalized data packet, and verify the transaction attribute information carried by the normalized data packet based on the transaction verification code, and determine the verification result;

[0052] The big data warehouse is used to receive the normalized data packet, and verify the transaction attribute information carried by the normalized data packet based on the transaction verification code, and determine the verification result, including at least:

[0053] The transaction verification code and the corresponding transaction attribute information pre-received by the big data warehouse and notified to the big data warehouse by the verification code generation unit are compared with the transaction verification code carried by the corresponding transaction attribute information to determine whether they are the same, and if they are the same, the verification is passed;

[0054] When the verification is passed, the big data warehouse allows the transaction attribute information in the normalized data packet to be stored;

[0055] When the verification fails, the big data warehouse extracts the transaction attribute information in the normalized data packet, and based on the transaction unit ID therein, enables the system to shield the collection of transaction attribute information of the transaction unit corresponding to the transaction unit ID within a first time period.

[0056] Preferably, it also includes a verification code generating unit.

[0057] Preferably, the system presets the first collection information replacement table including the IDs of all first transaction parties and their corresponding replacement IDs, as well as the transaction time replacement strategy;

[0058] The transaction time replacement strategy is used to indicate how to replace the transaction time.

[0059] Preferably, the transaction verification code can be generated by a verification code generation unit of the system for each transaction attribute information, and the generated transaction verification code is added to the transaction attribute information and stored in the third transaction attribute information of the transaction attribute information.

[0060] At the same time, the verification code generation unit informs the big data warehouse of the transaction verification code and the corresponding transaction attribute information.

[0061] Preferably, the information data packet normalization module is hosted on a dedicated server.

[0062] The present invention proposes a big data information collection and processing system and a corresponding method, which realizes information collection and management based on big data by performing data supervision and collection on the transaction behaviors of transaction nodes at the front line of financial transactions. Among them, the information data is collected by a three-part discrete collection method, and the transaction data is divided into the first transaction attribute information, the second transaction attribute information and the third transaction attribute information, which are mutually coordinated and have certain independent collection characteristics. The corresponding collection servers are respectively used for big data collection. At the same time, the collection information replacement based on risk avoidance and attack prevention is performed on a specific collection server, and the system big data security is considered in the process of executing big data transmission and system storage. Through the data normalization of the data collection and normalization module of the collection server, the big data is conditionally executed into the warehouse, thereby realizing the safe collection of big data and differentiated storage management. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a basic system structure diagram of the big data information collection and processing system shown in the present invention;

[0064] Figure 2It is a basic system structure diagram of the interconnection between the big data warehouse and the information data packet normalization module in the big data information collection and processing system shown in the present invention;

[0065] Figure 3 It is a basic system structure diagram of interconnection between various acquisition servers and transaction units in the big data information acquisition and processing system shown in the present invention;

[0066] Figure 4 It is a preferred embodiment of the interconnection between the first acquisition information replacement server and the first acquisition server in the big data information acquisition and processing system shown in the present invention;

[0067] Figure 5 It is a schematic diagram of a preferred display embodiment of the big data information collection and processing system operation method shown in the present invention. DETAILED DESCRIPTION

[0068] The following describes in detail several embodiments and beneficial effects of a big data information collection and processing system claimed in the present invention to facilitate a more detailed review and analysis of the present invention.

[0069] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0070] It should be clear that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0072] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0073] It should be understood that although the terms first, second, etc. may be used to describe the method and the corresponding device in the embodiments of the present invention, these keywords should not be limited to these terms. These terms are only used to distinguish the keywords from each other. For example, without departing from the scope of the embodiments of the present invention, the first normalized data packet, the first collection information replacement server, etc. may also be referred to as the second normalized data packet, the second collection information replacement server, and similarly, the second normalized data packet, the second collection information replacement server, etc. may also be referred to as the first normalized data packet, the first collection information replacement server.

[0074] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0075] As the instruction manual Figure 1-4 As shown in the instruction manual, Figure 1-4 One of the embodiments of a big data information collection and processing system and its specific internal module interconnection relationship claimed by the present invention, the system includes:

[0076] A plurality of transaction units, each of which is distributed at various locations of the financial information transaction system, performs financial information transactions, and generates transaction attribute information;

[0077] The transaction attribute information at least includes:

[0078] The first party to the transaction, the second party to the transaction, the transaction time, the transaction unit ID, the transaction settlement amount data, and the transaction verification code;

[0079] The transaction verification code is used to verify the transaction attribute information;

[0080] As a superimposable embodiment, the transaction verification code is used to verify the transaction attribute information, specifically: when each transaction unit executes a financial information transaction and generates transaction attribute information, a specific transaction verification code is bound to the transaction attribute information. The transaction verification code can be generated by the verification code generation unit of the system for each transaction attribute information, and the generated transaction verification code is added to the transaction attribute information and saved in the third transaction attribute information of the transaction attribute information. At the same time, the verification code generation unit informs the big data warehouse of the transaction verification code and the corresponding transaction attribute information. When executing information storage, the big data warehouse checks whether the transaction verification code carried by the transaction attribute information is consistent with the verification code of the transaction attribute information in the big data warehouse. When it is judged to be yes, the transaction attribute information is stored.

[0081] A first collection server, configured to collect only first transaction attribute information from each of the plurality of transaction units, and generate a first normalized data packet based on the first transaction attribute information;

[0082] The first transaction attribute information at least includes: the first transaction party and the transaction time;

[0083] A second collection server, configured to collect only second transaction attribute information from each of the plurality of transaction units, and generate a second normalized data packet based on the second transaction attribute information;

[0084] The second transaction attribute information at least includes: the second transaction party and the transaction unit ID;

[0085] a third collection server, configured to collect only third transaction attribute information from each of the plurality of transaction units, and generate a third normalized data packet based on the third transaction attribute information;

[0086] Wherein, the third transaction attribute information at least includes: transaction settlement amount data and transaction verification code;

[0087] The big data information collection and processing system also includes:

[0088] a first acquisition information replacement server, configured to replace all of the first transaction attribute information in the first normalized data packet, or replace part of the first transaction attribute information in the first normalized data packet, based on a system preset first acquisition information replacement table; and return the replaced first normalized data packet to the first acquisition server as an updated first normalized data packet of the original first normalized data packet;

[0089] As a superimposable embodiment, the first acquisition information replacement server is used to replace the first transaction attribute information in the first normalized data packet based on the system preset first acquisition information replacement table. Specifically, the first acquisition information replacement server obtains the first normalized data packet and obtains the first transaction attribute information from the first normalized data packet, replaces the first transaction party or transaction time therein, or replaces the first transaction party and transaction time therein based on the system preset first acquisition information replacement table, and recomposes the replaced first normalized data packet and returns it to the first acquisition server. The system preset first acquisition information replacement table includes the IDs of all first transaction parties and their corresponding replacement IDs, as well as the transaction time replacement strategy. The transaction time replacement strategy is used to indicate how to replace the transaction time. For example, as another superimposable embodiment, the system can set a trading time format such as 20210303-153011, where 20210303 represents the year, month, and day, that is, March 3, 2021, and 153011 represents the hours, minutes, and seconds of the transaction, that is, 15:30:11. The trading time replacement strategy can be: arrange the trading time format in reverse order from high to low, for example, arranging it as 110351-30301202, thereby replacing the original trading time 20210303-153011. Here, the trading time replacement strategy is only used as an example, and is not to be interpreted as a limitation or specification of the trading time replacement strategy.

[0090] An information data packet normalization module, the information data packet normalization module is used to combine the updated first normalized data packet, the second normalized data packet and the third normalized data packet into a normalized data packet;

[0091] The normalized data packet contains complete transaction attribute information of a single transaction of a single transaction unit;

[0092] The information data packet normalization module is also used to send the normalized data packet to a big data warehouse;

[0093] A big data warehouse, the big data warehouse is used to receive a normalized data packet, and verify the transaction attribute information carried by the normalized data packet based on the transaction verification code, and determine the verification result;

[0094] The big data warehouse is used to receive the normalized data packet, and verify the transaction attribute information carried by the normalized data packet based on the transaction verification code, and determine the verification result, including at least:

[0095] The transaction verification code and the corresponding transaction attribute information pre-received by the big data warehouse and notified to the big data warehouse by the verification code generation unit are compared with the transaction verification code carried by the corresponding transaction attribute information to determine whether they are the same, and if they are the same, the verification is passed;

[0096] When the verification is passed, the big data warehouse allows the transaction attribute information in the normalized data packet to be stored;

[0097] When the verification fails, the big data warehouse extracts the transaction attribute information in the normalized data packet, and based on the transaction unit ID therein, enables the system to shield the collection of transaction attribute information of the transaction unit corresponding to the transaction unit ID within a first time period.

[0098] As another superimposable embodiment, it further includes a verification code generating unit.

[0099] As another superimposable embodiment, the system presets a first collection information replacement table including the IDs of all first transaction parties and their corresponding replacement IDs, as well as a transaction time replacement strategy;

[0100] The transaction time replacement strategy is used to indicate how to replace the transaction time.

[0101] As another superimposable embodiment, the transaction verification code may be generated by the verification code generation unit of the system for each transaction attribute information, and the generated transaction verification code is added to the transaction attribute information and stored in the third transaction attribute information of the transaction attribute information.

[0102] At the same time, the verification code generation unit informs the big data warehouse of the transaction verification code and the corresponding transaction attribute information.

[0103] As another superimposable embodiment, the information data packet normalization module is hosted on a dedicated server.

[0104] At the same time, as the instructions Figure 5 As shown in the instruction manual, Figure 5 This is a schematic diagram of a preferred display embodiment of the operation method of the big data information collection and processing system shown in the present invention, and the method includes the following steps:

[0105] S102: operating each of the multiple transaction units, distributed at various locations of the financial information transaction system, to execute financial information transactions and generate transaction attribute information;

[0106] The transaction attribute information at least includes:

[0107] The first party to the transaction, the second party to the transaction, the transaction time, the transaction unit ID, the transaction settlement amount data, and the transaction verification code;

[0108] The transaction verification code is used to verify the transaction attribute information;

[0109] As a superimposable embodiment, the transaction verification code is used to verify the transaction attribute information, specifically: when each transaction unit executes a financial information transaction and generates transaction attribute information, a specific transaction verification code is bound to the transaction attribute information. The transaction verification code can be generated by the verification code generation unit of the system for each transaction attribute information, and the generated transaction verification code is added to the transaction attribute information and saved in the third transaction attribute information of the transaction attribute information. At the same time, the verification code generation unit informs the big data warehouse of the transaction verification code and the corresponding transaction attribute information. When executing information storage, the big data warehouse checks whether the transaction verification code carried by the transaction attribute information is consistent with the verification code of the transaction attribute information in the big data warehouse. When it is judged to be yes, the transaction attribute information is stored.

[0110] S104: operating the first collection server to collect only the first transaction attribute information from each of the multiple transaction units, and generating a first normalized data packet based on the first transaction attribute information;

[0111] The first transaction attribute information at least includes: the first transaction party and the transaction time;

[0112] S106: operating the second collection server to collect only the second transaction attribute information from each of the plurality of transaction units, and generating a second normalized data packet based on the second transaction attribute information;

[0113] The second transaction attribute information at least includes: the second transaction party and the transaction unit ID;

[0114] S108: operating the third collection server to collect only the third transaction attribute information from each of the plurality of transaction units, and generating a third normalized data packet based on the third transaction attribute information;

[0115] Wherein, the third transaction attribute information at least includes: transaction settlement amount data and transaction verification code;

[0116] S110: operating the first acquisition information replacement server to replace all of the first transaction attribute information in the first normalized data packet, or to replace part of the first transaction attribute information in the first normalized data packet based on a system preset first acquisition information replacement table; and returning the replaced first normalized data packet to the first acquisition server as an updated first normalized data packet of the original first normalized data packet;

[0117] As a superimposable embodiment, the first acquisition information replacement server is used to replace the first transaction attribute information in the first normalized data packet based on the system preset first acquisition information replacement table. Specifically, the first acquisition information replacement server obtains the first normalized data packet and obtains the first transaction attribute information from the first normalized data packet, replaces the first transaction party or transaction time therein, or replaces the first transaction party and transaction time therein based on the system preset first acquisition information replacement table, and recomposes the replaced first normalized data packet and returns it to the first acquisition server. The system preset first acquisition information replacement table includes the IDs of all first transaction parties and their corresponding replacement IDs, as well as the transaction time replacement strategy. The transaction time replacement strategy is used to indicate how to replace the transaction time. For example, as another superimposable embodiment, the system can set a trading time format such as 20210303-153011, where 20210303 represents the year, month, and day, that is, March 3, 2021, and 153011 represents the hours, minutes, and seconds of the transaction, that is, 15:30:11. The trading time replacement strategy can be: arrange the trading time format in reverse order from high to low, for example, arranging it as 110351-30301202, thereby replacing the original trading time 20210303-153011. Here, the trading time replacement strategy is only used as an example, and is not to be interpreted as a limitation or specification of the trading time replacement strategy.

[0118] S112: operating the information data packet normalization module to combine the updated first normalized data packet, the second normalized data packet, and the third normalized data packet into a normalized data packet;

[0119] The normalized data packet contains complete transaction attribute information of a single transaction of a single transaction unit;

[0120] The information data packet normalization module is also used to send the normalized data packet to a big data warehouse;

[0121] S114: operating the big data warehouse to receive the normalized data packet, and verifying the transaction attribute information carried in the normalized data packet based on the transaction verification code, and determining the verification result;

[0122] The big data warehouse is used to receive the normalized data packet, and verify the transaction attribute information carried by the normalized data packet based on the transaction verification code, and determine the verification result, including at least:

[0123] The transaction verification code and the corresponding transaction attribute information pre-received by the big data warehouse and notified to the big data warehouse by the verification code generation unit are compared with the transaction verification code carried by the corresponding transaction attribute information to determine whether they are the same, and if they are the same, the verification is passed;

[0124] When the verification is passed, the big data warehouse allows the transaction attribute information in the normalized data packet to be stored;

[0125] When the verification fails, the big data warehouse extracts the transaction attribute information in the normalized data packet, and based on the transaction unit ID therein, enables the system to shield the collection of transaction attribute information of the transaction unit corresponding to the transaction unit ID within a first time period.

[0126] As another superimposable embodiment, it also includes a verification code generating unit.

[0127] As another superimposable embodiment, the system presets a first collection information replacement table including all transaction first party IDs and their corresponding replacement IDs, as well as a transaction time replacement strategy;

[0128] The transaction time replacement strategy is used to indicate how to replace the transaction time.

[0129] As another superimposable embodiment, the transaction verification code may be generated by the verification code generation unit of the system for each transaction attribute information, and the generated transaction verification code is added to the transaction attribute information and stored in the third transaction attribute information of the transaction attribute information.

[0130] At the same time, the verification code generation unit informs the big data warehouse of the transaction verification code and the corresponding transaction attribute information.

[0131] As another superimposable embodiment, the information data packet normalization module is hosted on a dedicated server.

[0132] The present invention proposes a big data information collection and processing system and a corresponding method, which realizes information collection and management based on big data by performing data supervision and collection on the transaction behaviors of transaction nodes at the front line of financial transactions. Among them, the information data is collected by a three-part discrete collection method, and the transaction data is divided into the first transaction attribute information, the second transaction attribute information and the third transaction attribute information, which are mutually coordinated and have certain independent collection characteristics. The corresponding collection servers are respectively used for big data collection. At the same time, the collection information replacement based on risk avoidance and attack prevention is performed on a specific collection server, and the system big data security is considered in the process of executing big data transmission and system storage. Through the data normalization of the data collection and normalization module of the collection server, the big data is conditionally executed into the warehouse, thereby realizing the safe collection of big data and differentiated storage management.

[0133] In all the above embodiments, in order to realize some special data transmission and read / write function requirements, the above method operation process and its corresponding device can add devices, modules, components, hardware, pin connections or memory, processor differences to expand functions.

[0134] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the methods, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0135] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the method steps is only a logical or functional division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0136] The units described as the various steps of the method and the separated components of the device may or may not be logically or physically separated, and may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] In addition, the method steps and their implementations and functional units in the various embodiments of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0138] The above method and device can be implemented as an integrated unit in the form of a software functional unit, which can be stored in a computer-readable storage medium. The above software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), NVRAM, disk or optical disk, etc., which can store program code.

[0139] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0140] It should be noted that the above embodiments are only used to more clearly explain and illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A big data information collection and processing system, the system comprising: A plurality of transaction units, each of which is distributed at various locations of the financial information transaction system, performs financial information transactions, and generates transaction attribute information; The transaction attribute information at least includes: The first party to the transaction, the second party to the transaction, the transaction time, the transaction unit ID, the transaction settlement amount data, and the transaction verification code; The transaction verification code is used to verify the transaction attribute information; A first collection server, configured to collect only first transaction attribute information from each of the plurality of transaction units, and generate a first normalized data packet based on the first transaction attribute information; The first transaction attribute information at least includes: the first transaction party and the transaction time; A second collection server, configured to collect only second transaction attribute information from each of the plurality of transaction units, and generate a second normalized data packet based on the second transaction attribute information; The second transaction attribute information at least includes: the second transaction party and the transaction unit ID; a third collection server, configured to collect only third transaction attribute information from each of the plurality of transaction units, and generate a third normalized data packet based on the third transaction attribute information; Wherein, the third transaction attribute information at least includes: transaction settlement amount data and transaction verification code; The big data information collection and processing system also includes: a first acquisition information replacement server, configured to replace all of the first transaction attribute information in the first normalized data packet, or replace part of the first transaction attribute information in the first normalized data packet, based on a system preset first acquisition information replacement table; and return the replaced first normalized data packet to the first acquisition server as an updated first normalized data packet of the original first normalized data packet; An information data packet normalization module, the information data packet normalization module is used to combine the updated first normalized data packet, the second normalized data packet and the third normalized data packet into a normalized data packet; The normalized data packet contains complete transaction attribute information of a single transaction of a single transaction unit; The information data packet normalization module is also used to send the normalized data packet to a big data warehouse; A big data warehouse, the big data warehouse is used to receive a normalized data packet, and verify the transaction attribute information carried by the normalized data packet based on the transaction verification code, and determine the verification result, including at least: The transaction verification code and the corresponding transaction attribute information pre-received by the big data warehouse and notified to the big data warehouse by the verification code generation unit are compared with the transaction verification code carried by the corresponding transaction attribute information to determine whether they are the same, and if they are the same, the verification is passed; When the verification is passed, the big data warehouse allows the transaction attribute information in the normalized data packet to be stored; When the verification fails, the big data warehouse extracts the transaction attribute information in the normalized data packet, and based on the transaction unit ID therein, enables the system to shield the collection of transaction attribute information of the transaction unit corresponding to the transaction unit ID within the first time period; The big data information collection and processing system also includes a verification code generation unit; The system presets the first collection information replacement table including the IDs of all first transaction parties and their corresponding replacement IDs, as well as the transaction time replacement strategy; The trading time replacement strategy is used to indicate how to replace the trading time; The transaction verification code is generated by the system's verification code generation unit for each transaction attribute information, and the generated transaction verification code is added to the transaction attribute information and saved in the third transaction attribute information of the transaction attribute information. At the same time, the verification code generation unit informs the big data warehouse of the transaction verification code and the corresponding transaction attribute information.

2. A big data information collection and processing system as claimed in claim 1, characterized in that: The information data packet normalization module is carried on a dedicated server.

3. A method for operating a big data information collection and processing system, the method comprising the following steps: Step 1: operate each of the multiple transaction units, distributed at various locations of the financial information transaction system, execute financial information transactions, and generate transaction attribute information; The transaction attribute information at least includes: The first party to the transaction, the second party to the transaction, the transaction time, the transaction unit ID, the transaction settlement amount data, and the transaction verification code; The transaction verification code is used to verify the transaction attribute information; Step 2: operating the first collection server to collect only the first transaction attribute information from each of the multiple transaction units, and generating a first normalized data packet based on the first transaction attribute information; The first transaction attribute information at least includes: the first transaction party and the transaction time; Step 3: operating the second collection server to collect only the second transaction attribute information from each of the multiple transaction units, and generating a second normalized data packet based on the second transaction attribute information; The second transaction attribute information at least includes: the second transaction party and the transaction unit ID; Step 4: operate the third collection server to collect only the third transaction attribute information from each of the multiple transaction units, and generate a third normalized data packet based on the third transaction attribute information; wherein the third transaction attribute information at least includes: transaction settlement amount data and a transaction verification code; Step 5: operate the first acquisition information replacement server to replace all of the first transaction attribute information in the first normalized data packet, or replace part of the first transaction attribute information in the first normalized data packet based on the system preset first acquisition information replacement table; and return the replaced first normalized data packet to the first acquisition server as an updated first normalized data packet of the original first normalized data packet; Step 6: operate the information data packet normalization module to combine the updated first normalized data packet, the second normalized data packet and the third normalized data packet into a normalized data packet; The normalized data packet contains complete transaction attribute information of a single transaction of a single transaction unit; The information data packet normalization module is also used to send the normalized data packet to a big data warehouse; Step 7: Operate the big data warehouse to receive the normalized data packet, and verify the transaction attribute information carried in the normalized data packet based on the transaction verification code, and determine the verification result, which at least includes: The transaction verification code and corresponding transaction attribute information pre-received by the big data warehouse and notified to the big data warehouse by the verification code generation unit; Compare the transaction verification code with the corresponding transaction attribute information to determine whether they are the same, if they are the same, the verification is passed; When the verification is passed, the big data warehouse allows the transaction attribute information in the normalized data packet to be stored; When the verification fails, the big data warehouse extracts the transaction attribute information in the normalized data packet, and based on the transaction unit ID therein, enables the system to shield the collection of transaction attribute information of the transaction unit corresponding to the transaction unit ID within the first time period; Also includes a verification code generating unit; The preset first collection information replacement table includes the IDs of all first transaction parties and their corresponding replacement IDs, as well as the transaction time replacement strategy; The trading time replacement strategy is used to indicate how to replace the trading time; The transaction verification code is generated by the system's verification code generation unit for each transaction attribute information, and the generated transaction verification code is added to the transaction attribute information and saved in the third transaction attribute information of the transaction attribute information. At the same time, the verification code generation unit informs the big data warehouse of the transaction verification code and the corresponding transaction attribute information.

4. The operating method according to claim 3, characterized in that: The information data packet normalization module is carried on a dedicated server.

Images