Multi-channel data processing method and device, medium and program product

Abstract

The embodiment of the invention discloses a multi-channel data processing method and device, a medium and a program product. The method comprises the following steps: continuously receiving target multi-channel data from multiple channels through an application programming interface, and pushing each piece of target multi-channel data to a message queue; determining the creation time of the target multi-channel data, and if the creation time is within the adjacent time period, processing the target multi-channel data in the relational database; the relational database stores multi-channel data of projects with creation time in adjacent time periods; and if the creation time is within the long-term time period, processing the target multi-channel data in the non-relational database. According to the scheme, real-time receiving and successful storage can be carried out on the multi-channel high-concurrency multi-channel data, and the problems of receiving blockage, storage delay and even failure of the multi-channel high-concurrency multi-channel data at present are solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a multi-channel data processing method, device, medium and program product. Background Art

[0002] With the advancement of the digital age, online products of the same type are flourishing. Even on the same APP platform, there are multiple online products of the same type. User transaction data is scattered across multiple product channels. With the gradual promotion of onlineization, there are more and more high-concurrency scenarios and the amount of transaction data is also increasing. This has brought new challenges to data storage. Even for similar products in different channels, integration is very difficult due to inconsistent data models and different business processes. At present, the unified view display device for multi-channel heterogeneous data is: directly persisted to the relational database through an online program, and then uniformly displayed by querying the relational database data, unifying the subsequent business processing process, and returning the user's change operations in the unified view to each channel to maintain data consistency. This technical solution has the following disadvantages: (1) Direct persistence to the relational database lacks a buffer link and cannot cope with high-concurrency data synchronization scenarios. (2) With the increasing amount of synchronized data, the relational database cannot support the storage of large amounts of data. (3) The relational database cannot support high-concurrency query scenarios for large amounts of data. (4) The device needs to transmit transaction change data back to each channel. Each additional channel requires customized development of the relevant feedback interface, which is highly complex and requires a lot of work to implement. Summary of the Invention

[0003] The embodiments of the present application provide a multi-channel data processing method, device, medium and program product to achieve real-time reception, caching and persistent storage of highly concurrent multi-channel data from multiple channel sources.

[0004] According to one aspect of the present application, a multi-channel data processing method is provided, the method comprising:

[0005] Continuously receive target multi-channel data from multiple channels through the application programming interface, and push each target multi-channel data to the message queue;

[0006] Reading the target multi-channel data one by one from the message queue, and determining the creation time of the item corresponding to the target multi-channel data;

[0007] If the creation time is within a contiguous time period, processing the target multi-channel data in a relational database; wherein the contiguous time period is a time period before the current time and less than a preset time interval from the current time; the relational database stores the multi-channel data of the project whose creation time is within the contiguous time period;

[0008] If the creation time is within a distant time period, the target multi-channel data is processed in a non-relational database; wherein the distant time period is a time period before the adjacent time period, and the non-relational database stores multi-channel data of projects whose creation time is within the distant time period.

[0009] According to one aspect of the present application, a multi-channel data processing device is provided, the device comprising:

[0010] A push module is used to continuously receive target multi-channel data from multiple channels through an application programming interface, and push each target multi-channel data to a message queue;

[0011] a reading module, configured to read the target multi-channel data one by one from the message queue, and determine the creation time of the item corresponding to the target multi-channel data;

[0012] a first processing module configured to process the target multi-channel data in a relational database if the creation time is within a proximal time period; wherein the proximal time period is a time period before the current time and less than a preset time interval from the current time; and wherein the relational database stores the multi-channel data of the project whose creation time is within the proximal time period;

[0013] The second processing module is used to process the target multi-channel data in a non-relational database if the creation time is within a distant time period; wherein the distant time period is a time period before the adjacent time period, and the non-relational database stores multi-channel data of projects whose creation time is within the distant time period.

[0014] According to another aspect of the present application, an electronic device is provided, the electronic device comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by at least one processor so that the at least one processor can execute the multi-channel data processing method of any embodiment of the present application.

[0018] According to another aspect of the present application, a computer-readable storage medium is provided, which stores computer instructions, and the computer instructions are used to enable a processor to implement the multi-channel data processing method of any embodiment of the present application when executed.

[0019] According to another aspect of the present application, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the multi-channel data processing method of any embodiment of the present application is implemented.

[0020] The technical solution of the embodiment of the present application continuously receives target multi-channel data from multiple channels through an application programming interface, pushes each target multi-channel data into a message queue, and can use the message queue to quickly receive the stored data to achieve rapid caching of the target multi-channel data, solving the problem of receiving-end congestion caused by long storage time when directly storing in high-concurrency scenarios. The target multi-channel data is read one by one from the message queue to determine the creation time of the project corresponding to the target multi-channel data; if the creation time is within a nearby time period, the target multi-channel data is processed in a relational database; if the creation time is within a distant time period, the target multi-channel data is processed in a non-relational database, thereby achieving gradual and orderly storage in the database, solving the problem of storage failure caused by limited space in a distributed database and difficulty in accommodating continuously received target multi-channel data.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] Figure 1 A flowchart of a multi-channel data processing method provided in an embodiment of the present application;

[0024] Figure 2 A flowchart of a multi-channel data processing method provided in another embodiment of the present application;

[0025] Figure 3 A flowchart of a multi-channel data processing method provided in another embodiment of the present application;

[0026] Figure 4 A flowchart of a specific implementation method for tracking provided in an embodiment of the present application;

[0027] Figure 5 A schematic diagram of the structure of a multi-channel data processing device provided in an embodiment of the present application;

[0028] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0030] It should be noted that the terms "first", "second", "third", "fourth", "actual", "preset", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Figure 1 This is a flowchart of a multi-channel data processing method provided in an embodiment of the present application. This embodiment of the present application is applicable to the situation where high-concurrency target multi-channel data from multiple channels is received and stored. The method can be executed by a multi-channel data processing device, which can be implemented in the form of hardware and / or software, and the multi-channel data processing device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0032] S110 , continuously receiving target multi-channel data from multiple channels through an application programming interface, and pushing each target multi-channel data to a message queue.

[0033] The application programming interface (API) can be a set of predefined functions that process received data or requests. When user data is generated across multiple channels, the API of the multi-channel data processing device is called to receive multi-channel data transmitted by each channel. Multi-channels refer to multiple different platforms, devices, servers, etc. that can generate data, and target multi-channel data refers to data from multiple different channels. A message queue is a linked list or container that stores messages, temporarily storing data during message transmission and supporting asynchronous communication between different systems or components.

[0034] Exemplarily, the multi-channel data processing device may be connected to multiple channels to receive data generated by multiple channels, namely, target multi-channel data. Generally, after receiving the target multi-channel data, the target multi-channel data needs to be persistently stored. However, in some high-concurrency scenarios, multiple channels generate a large amount of target multi-channel data at the same time, and the efficiency of persistent storage is limited. Writing to the database takes a certain amount of time, which may cause a large amount of high-concurrency target multi-channel data to be unable to be persistently stored in time, affecting the application programming interface's continuous reception of the target multi-channel data, and may cause reception congestion or even reception failure. In an embodiment of the present application, in the process of the application programming interface continuously receiving target multi-channel data from multiple channels, each target multi-channel data can be pushed to the message queue for caching first, so as to avoid a large amount of target multi-channel data from simultaneously impacting the back-end storage service, resulting in the loss of target multi-channel data when the back-end storage service fails to respond.

[0035] S120: Read the target multi-channel data one by one from the message queue, and determine the creation time of the item corresponding to the target multi-channel data.

[0036] For example, the target multi-channel data stored in the message queue is read from the message queue one by one, and then persistently stored in the database one by one. Because the target multi-channel data is cached in the message queue and then gradually persisted in the database, there is no situation where a large amount of target multi-channel data is stored at the same time, which reduces the impact on the backend storage service, can achieve orderly storage of the target multi-channel data, and will not cause the loss of the target multi-channel data.

[0037] In an embodiment of the present application, before the target multi-channel data is processed, it is necessary to determine the creation time of the target multi-channel data corresponding to the same project, so as to determine whether the target multi-channel data needs to be processed in a relational database or a non-relational database. The same project of the target multi-channel data is a project generated for processing an event that the user needs to complete in the channel, and may include data at different stages. For example, the same project may be a transaction order of the user, and different target multi-channel data may be generated at different stages of the transaction. For example, different target multi-channel data may be generated in the initial order stage, the order modification node, the order completion stage, etc., but each target multi-channel data belongs to the same project, such as the same transaction order of the user.

[0038] S130. If the creation time is within an adjacent time period, the target multi-channel data is processed in a relational database; wherein the adjacent time period is a time period before the current time and less than a preset time interval from the current time; the relational database stores multi-channel data of projects whose creation time is within the adjacent time period.

[0039] The adjacent time period may be a time period before the current time and less than a preset time interval from the current time. For example, if the current time corresponds to date T and the preset time interval is N days, the adjacent time period is the time period between date T N and date T. Multi-channel data of projects whose creation times fall within the adjacent time period is stored in a relational database.

[0040] For example, if the creation time of the project corresponding to the target multi-channel data is within an adjacent time period, the multi-channel data of the project should be stored in a relational database. Regardless of whether the target multi-channel data is newly created data or data that needs to update previous multi-channel data, the target multi-channel data needs to be processed in the relational database.

[0041] S140. If the creation time is within a distant time period, the target multi-channel data is processed in a non-relational database; wherein the distant time period is a time period before the adjacent time period, and the non-relational database stores multi-channel data of projects whose creation time is within the distant time period.

[0042] The distant time period is the time period before the adjacent time period. For example, if the adjacent time period is the time period between TN and T, the distant time period is the time period before TN. Multi-channel data of projects with creation times in the distant time period is stored in a non-relational database.

[0043] For example, if the creation time of the project corresponding to the target multi-channel data is in the distant time period, it reflects that the multi-channel data corresponding to the project has been stored in the non-relational database, and therefore the target multi-channel data is processed in the non-relational database.

[0044] The technical solution of the embodiment of the present application continuously receives target multi-channel data from multiple channels through an application programming interface, pushes each target multi-channel data into a message queue, and can use the message queue to quickly receive the stored data to achieve rapid caching of the target multi-channel data, solving the problem of receiving-end congestion caused by long storage time when directly storing in high-concurrency scenarios. The target multi-channel data is read one by one from the message queue to determine the creation time of the project corresponding to the target multi-channel data; if the creation time is within a nearby time period, the target multi-channel data is processed in a relational database; if the creation time is within a distant time period, the target multi-channel data is processed in a non-relational database, thereby achieving gradual and orderly storage in the database, solving the problem of storage failure caused by limited space in a distributed database and difficulty in accommodating continuously received target multi-channel data.

[0045] As a non-limiting implementation, before continuously receiving target multi-channel data from multiple channels through the application programming interface, the method further includes:

[0046] Collect user data from multiple channels and count each field in each user data;

[0047] Aggregate all fields in all user data and create a unified data model that includes all fields;

[0048] Accordingly, after continuously receiving target multi-channel data from multiple channels through the application programming interface, the method further includes:

[0049] The value of each field in the target multi-channel data is filled into the corresponding field in the unified data model to form target multi-channel data that conforms to a unified format.

[0050] Exemplarily, user data from multiple channels can be collected in advance, and each field in each user data can be counted, that is, each type of information contained in the user data can be counted. In the embodiment of the present application, user data, multi-channel data, etc. are all data obtained through legal channels and with the user's consent. In fact, they do not include the user's personal privacy data, but only relevant information generated for events requested by the user, such as relevant information about transaction orders. All fields contained in all user data are aggregated to create a unified data model containing all fields. For example, the fields contained in user data a are user name, transaction goods, order number, and transaction amount, and the fields contained in user data b are user nickname, transaction goods, transaction amount, order number, and transaction time. The fields in user data a and user data b are uniformly aggregated to obtain a unified data model, that is, the same data model containing user name, user nickname, transaction goods, order number, transaction amount, and transaction time. The unified data model can only contain each field, and the values therein are replaced by placeholders.

[0051] Accordingly, after continuously receiving target multi-channel data from multiple channels through the application programming interface, the values of each field in the target multi-channel data can be filled into the corresponding fields in the unified data model to form target multi-channel data in a unified format, which is convenient for subsequent unified parsing and processing.

[0052] Figure 2 This is a flowchart of a multi-channel data processing method provided by another embodiment of the present application. The present embodiment is optimized based on the above embodiment. For solutions not fully described in the present embodiment, please refer to the above embodiment. Figure 2 As shown, the method of the embodiment of the present application specifically includes the following steps:

[0053] S210 , continuously receiving target multi-channel data from multiple channels through an application programming interface, and pushing each target multi-channel data to a message queue.

[0054] S220: Read the target multi-channel data one by one from the message queue, and determine the creation time of the item corresponding to the target multi-channel data.

[0055] S230: Determine a processing type for the target multi-channel data according to a type of an application programming interface for receiving the target multi-channel data; wherein the processing type includes a new creation type and a change type.

[0056] Exemplarily, when each channel calls an API for the generated target multi-channel data, it adaptively selects an appropriate API based on the processing type of the target multi-channel data, using the API corresponding to the processing type. For example, if the processing type of the target multi-channel data is a new creation type, the API corresponding to the new creation type is called for reception; if the processing type of the target multi-channel data is a change type, the API corresponding to the change type is called for reception.

[0057] S240: If the creation time is within an adjacent time period and the processing type of the target multi-channel data is a new creation type, the target multi-channel data is stored in a relational database.

[0058] For example, if the creation time of the project corresponding to the target multi-channel data falls within a contiguous time period, processing is required in the relational database. If the processing type of the target multi-channel data is new, that is, a new project is created, the target multi-channel data is directly stored in the relational database, and the target multi-channel data newly created within the contiguous time period is stored in the relational database.

[0059] S250: If the processing type of the target multi-channel data is a change type, query the historical multi-channel data corresponding to the target multi-channel data from the relational database, and change the historical multi-channel data based on the target multi-channel data.

[0060] For example, if the creation time of the project corresponding to the target multi-channel data is within a contiguous time period, and if the processing type of the target multi-channel data is a change type, meaning that the previously stored historical multi-channel data corresponding to the project needs to be modified, the historical multi-channel data corresponding to the target multi-channel data is queried from the relational database, that is, the historical multi-channel data belonging to the same project as the target multi-channel data is queried. The historical multi-channel data is modified based on the target multi-channel data.

[0061] S260: If the processing type of the target multi-channel data is a change type, query the historical multi-channel data corresponding to the target multi-channel data from the non-relational database, and change the historical multi-channel data based on the target multi-channel data.

[0062] For example, if the creation time of the project corresponding to the target multi-channel data falls within a distant time period, the target multi-channel data needs to be processed in a non-relational database. Specifically, the non-relational database is queried for historical multi-channel data corresponding to the target multi-channel data, that is, historical multi-channel data belonging to the same project as the target multi-channel data. The historical multi-channel data is then modified based on the target multi-channel data.

[0063] Specifically, the process of changing the historical multi-channel data based on the target multi-channel data may be to change the values of some fields in the historical multi-channel data, for example, changing the status of a user transaction order from "pending payment" to "payment completed".

[0064] An embodiment of the present application provides a multi-channel data processing method, which determines the processing type of the target multi-channel data according to the type of the application programming interface that receives the target multi-channel data. If the creation time is within a nearby time period and the processing type of the target multi-channel data is a new type, the target multi-channel data is stored in a relational database; if the processing type of the target multi-channel data is a change type, the historical multi-channel data corresponding to the target multi-channel data is queried from the relational database, and the historical multi-channel data is changed based on the target multi-channel data. If the creation time is within a distant time period and the processing type of the target multi-channel data is a change type, the historical multi-channel data corresponding to the target multi-channel data is queried from the non-relational database, and the historical multi-channel data is changed based on the target multi-channel data. The above scheme can adaptively determine the processing method of the target multi-channel data based on the storage characteristics of relational databases and non-relational databases, according to the creation time of the project corresponding to the target multi-channel data and the processing type of the target multi-channel data, thereby realizing orderly storage of the target multi-channel data.

[0065] As a non-limiting implementation, the method further includes:

[0066] Detecting historical multi-channel data within a preset time period with the earliest creation time in the relational database at a preset frequency; wherein the preset time period is the time period that first falls within the long-term time period;

[0067] The historical multi-channel data is constructed into a combination of a master table and a detail table, and converted into a preset format and stored in a temporary table; wherein the master table contains data of preset fields in the historical channel data; and the detail table contains other data in the historical channel data except the preset fields;

[0068] The historical multi-channel data in the temporary table is separated into various fields according to a preset separator and stored in a non-relational database.

[0069] The preset frequency can be determined based on actual circumstances. For example, if the unit is daily, the preset frequency can be once a day. The preset time period is the time period that first falls into the distant time period as time passes. For example, if the period from TN to T is a nearby time period and the period before TN is a distant time period, when T+1 arrives, TN, which is within the nearby time period, enters the distant time period, and TN is then used as the preset time period.

[0070] Exemplarily, historical multi-channel data within a preset time period with the earliest creation time is obtained from a relational database, and the historical multi-channel data is constructed in the form of a combination of a main table and a detailed table. The main table contains data of preset fields in the historical channel data, which are generally the main fields that users often use and pay attention to, such as user name, product name, product amount, transaction time, etc. The detailed table contains other data in the historical channel data except the preset fields, such as product return and exchange status, evaluation information, freight information, after-sales rights information, etc. Storing in the form of a combination of a main table and a detailed table enables more convenient and rapid step-by-step retrieval when stored in a non-relational database. The historical multi-channel data constructed in the form of a combination of a main table and a detailed table is converted into a preset format and stored in a temporary table. The preset format can be a format supported by the non-relational database. The historical multi-channel data is read from the temporary table and each field is separated by a preset delimiter and stored in a non-relational database.

[0071] Figure 3 This is a flowchart of a multi-channel data processing method provided by another embodiment of the present application. The present embodiment is optimized based on the above embodiment. For solutions not fully described in the present embodiment, please refer to the above embodiment. Figure 3 As shown, the method of the embodiment of the present application specifically includes the following steps:

[0072] S310 , continuously receiving target multi-channel data from multiple channels through an application programming interface, and pushing each target multi-channel data to a message queue.

[0073] S320: Read the target multi-channel data one by one from the message queue, and determine the creation time of the item corresponding to the target multi-channel data.

[0074] S330. If the creation time is within an adjacent time period, the target multi-channel data is processed in a relational database; wherein the adjacent time period is a time period before the current time and less than a preset time interval from the current time; the relational database stores multi-channel data of projects whose creation time is within the adjacent time period.

[0075] S340. If the creation time is within an adjacent time period, the target multi-channel data is processed in a relational database; wherein the adjacent time period is a time period before the current time and less than a preset time interval from the current time; the relational database stores multi-channel data of projects whose creation time is within the adjacent time period.

[0076] S350: Obtain the user's query conditions through the front-end page, parse and obtain the filter conditions, the result display page number, and the number of results displayed on a single page.

[0077] For example, after storing the target multi-channel data, users can perform queries based on their actual needs. Users can enter query criteria on the front-end page, which is then parsed to obtain the filter criteria, the result display page number, and the number of results displayed per page. The filter criteria reflect the data the user is searching for, the result display page number reflects the number of pages containing the query results the user is currently viewing, and the number of results displayed per page is the number of query results displayed on a single page.

[0078] S360: Search the relational database and / or non-relational database based on the filtering conditions, the result display page number, and the number of result items displayed on a single page, and display the returned result multi-channel data on the front-end page.

[0079] For example, a search can be performed in a relational database and / or a non-relational database based on the filtering conditions, the result display page number, and the number of results displayed on a single page, so as to find the results that meet the filtering conditions and the requirements for display on the page, and the returned result multi-channel data is displayed on the front-end page.

[0080] In the embodiment of the present application, searching in the relational database and / or non-relational database according to the screening conditions, the result display page number, and the number of result items displayed on a single page includes:

[0081] For the current page selected by the user for viewing, determine the target number of results to be displayed based on the total number of pages from the first page to the current page and the number of results displayed on a single page;

[0082] If the number of results obtained from the query of the relational database based on the screening condition is greater than or equal to the target number of results, the multi-channel data of the results obtained from the query of the relational database that meet the target number of results and are ranked first are displayed on the front-end page;

[0083] Otherwise, continue to query the remaining number of result multi-channel data from the non-relational database based on the screening conditions, and display the result multi-channel data that meets the target number of result items queried from the relational database and the non-relational database on the front-end page; wherein the remaining number is the difference between the target number of result items and the number of result items queried from the relational database.

[0084] For example, in the process of displaying search results on the front-end page, the current page selected by the user for viewing can be determined first, and the target number of results to be displayed can be determined based on the total number of pages from the first page to the current page and the number of results displayed on a single page. For example, if the user chooses to view page 4 and 10 results are displayed on each page, a total of 40 multi-channel data results need to be queried. A query can be performed from the relational database first. If the number of results obtained from the query in the relational database is greater than or equal to the target number of results, and the display of results from the first page to the current page is satisfied, the multi-channel data of the results that meet the target number of results and are ranked first are directly read from the relational database for display on the front-end page. If the result obtained from the query in the relational database is less than the target number of results and does not meet the requirement for displaying the results from the first page to the current page, a supplementary query is required from the non-relational database, and the multi-channel data of the remaining results are queried from the non-relational database according to the filtering conditions. The front-end page displays all the multi-channel data of the results obtained from the relational database and the multi-channel data of the remaining results obtained from the non-relational database, and displays the multi-channel data of the target number of results in total, meeting the user's requirements for the result display page number and the number of results displayed on a single page.

[0085] In an embodiment of the present application, the method further includes:

[0086] Displaying a list of master data in the multi-channel data obtained by querying the user's query conditions on the front-end page; wherein the master data list is a list of data corresponding to the preset fields in the query results;

[0087] A detail page jump link is set for each master data list so that when the user clicks to view the detail page, the link will jump to the channel to which the master data list belongs for detailed data query.

[0088] For example, when displaying multi-channel data obtained based on the user's query criteria on the front-end page, a master data list is displayed, that is, the data corresponding to the preset fields are displayed in a list to present the main fields more concisely and intuitively. A jump link to the details page is set for each master data list. When the user clicks to view the details page, the user jumps to the channel to which the master data list belongs, and then jumps to the corresponding channel platform to view detailed data.

[0089] An embodiment of the present application provides a multi-channel data processing method, which obtains the user's query conditions through a front-end page, parses them to obtain filtering conditions, result display page numbers, and the number of results displayed on a single page; based on the filtering conditions, result display page numbers, and the number of results displayed on a single page, searches are performed in the relational database and / or non-relational database, and the returned multi-channel data is displayed on the front-end page, thereby realizing joint reading from the relational database and the non-relational database, and realizing a complete query of the data.

[0090] The embodiment of the present application provides a specific implementation method, the overall process is as follows Figure 4 As shown,

[0091] 1. The homogeneous heterogeneous data synchronization and integration unit synchronizes and integrates data, buffers target multi-channel data through message queues, and processes target multi-channel data in relational and non-relational databases: It is necessary to manually collect user transaction data models from several channels in advance, analyze user transaction data objects, aggregate all fields to build a unified data model, and create relevant table structures in the relational database.

[0092] (1) When a user conducts a related transaction on the channel side, the channel calls the data synchronization transaction interface online to synchronize the user's transaction data. After the heterogeneous data synchronization integration unit receives the call request, the gateway layer first verifies the identity of the channel side and the integrity of the message, and also provides a flow control function. After the verification is passed, the data synchronization API interface is called.

[0093] (2) The API interface converts the user transaction data in the request message into a data model and then pushes it to the message queue.

[0094] (3) The persistent monitoring program adopts a clustered deployment method, with multiple nodes reading the data in the message queue in real time. If a new user transaction is added, it will be directly persisted to the relational database. If the user transaction data is changed, and the creation time of the user transaction data is greater than or equal to TN, that is, the data is created after TN, then the data will be queried and the data will be changed. If the creation time of the user transaction data is less than TN, that is, the data is created before TN, then the "online synchronization module" in the "historical data processing unit" will be called to persist the data change directly to the distributed non-relational database.

[0095] 2. Historical data processing unit: This unit includes 3 modules: batch synchronization module, online synchronization module, and historical data query module.

[0096] (1) Batch synchronization module:

[0097] Step 1: The job centralized scheduling platform queries the user transaction data of the (T - N)th day from the relational database at the end of each day.

[0098] Step 2: Each piece of data corresponds to a customer's transaction data. For data with a one - to - many relationship between the main table and the detail table, multiple detail data are converted into JSON format and stored as a column in the temporary table.

[0099] Step 3: Query the user transaction data of the (T - N)th day from the temporary table and output the corresponding data to a file using the specified delimiter.

[0100] Step 4: Import the file generated in Step 3 into the non - relational database for storage.

[0101] Step 5: After Step 4 is successfully executed, clean up the user data of the (T - N)th day in the relational database to solve the problem of the increasing storage capacity of the relational database.

[0102] (2) Online synchronization module: For data before (T - N) days, if a piece of data has changed, the API in the "homogeneous heterogeneous data synchronization and integration unit" will call this module to update the corresponding data in the distributed non - relational database.

[0103] (3) Historical data query module: According to the input conditions, query the data that meets the conditions in the distributed non - relational database and return the corresponding row number r at the same time.

[0104] 3. Unified view display unit:

[0105] (1) The user enters the query conditions, and the page encapsulates information such as the filtering conditions, page number m, page size s, etc. into the message and requests the backend query interface.

[0106] (2) The backend query interface first queries the data in the relational database. If the number of returned data rows n is equal to s, it directly returns; if it is less than s, it queries (s - n) data from the distributed non - relational database.

[0107] Step 1: When the user queries the first page, first query the total number of data rows n that meet the conditions from the relational database. If the number of rows n >= s * m and at this time m = 1, then query the first s data rows from the relational database and return; if n < s, then query (s - n) data from the distributed non - relational database and return the row number r of the (s - n)th data row.

[0108] Step 2: The user queries the second page. If the row number r > 0, that is, the first page has been queried from the distributed non-relational database, then this query directly starts from the (r + 1)-th row data of the non-relational database and returns the queried data. If r = 0, then in the way of Step 1, first query from the relational database. If the number of rows n >= s * m, where m = 2 at this time, then query the first s rows of data in the relational database and return them; if n < s, then query s - n rows of data from the distributed non-relational database and return the row number r of the (s - n)-th row of data.

[0109] Step 3: Query the third, fourth... pages in the way of Step 2. If the number of rows of data on the queried current page < s, then there is no more data.

[0110] Page display: Render the data queried from the backend interface on the page, and cache the data queried for each page. When the user flips the page in reverse order, there is no need to query the backend interface again, which reduces the pressure on the backend and also speeds up the response speed of the page. The unified view only displays the main file data list, supports comprehensive sorting and searching, and jumps to each channel when clicking to query details, and performs relevant business processes in each channel.

[0111] Figure 5 It is a schematic structural diagram of a multi-channel data processing device provided by an embodiment of the present application. This device can execute the multi-channel data processing method provided by any embodiment of the present application, and has corresponding functional modules and beneficial effects for executing the method. As Figure 5 shown, the device includes:

[0112] A push module 410, configured to continuously receive target multi-channel data from multiple channels through an application programming interface, and push each target multi-channel data into a message queue;

[0113] A reading module 420, configured to read the target multi-channel data one by one from the message queue and determine the creation time of the project corresponding to the target multi-channel data;

[0114] A first processing module 430, configured to, if the creation time is within an adjacent time period, process the target multi-channel data in a relational database; wherein, the adjacent time period is a time period before the current time and less than a preset time interval from the current time; the relational database stores multi-channel data of projects with creation times within the adjacent time period;

[0115] A second processing module 440, configured to, if the creation time is within a long-term time period, process the target multi-channel data in a non-relational database; wherein, the long-term time period is a time period before the adjacent time period, and the non-relational database stores multi-channel data of projects with creation times within the long-term time period.

[0116] In the embodiment of the present application, before continuously receiving target multi-channel data from multiple channels through the application programming interface, the apparatus further includes:

[0117] The collection module is used to collect user data from multiple channels and count each field in each user data;

[0118] A unified data model creation module is used to aggregate all fields in all user data and create a unified data model containing all fields;

[0119] Accordingly, after continuously receiving target multi-channel data from multiple channels through the application programming interface, the apparatus further includes:

[0120] The filling module is used to fill the value of each field in the target multi-channel data into the corresponding field in the unified data model to form the target multi-channel data in a unified format.

[0121] In an embodiment of the present application, the device further includes:

[0122] a processing type determining module, configured to determine a processing type of the target multi-channel data according to a type of an application programming interface for receiving the target multi-channel data; wherein the processing type includes a new creation type and a change type;

[0123] If the creation time is within the adjacent time period, the first processing module 430 processes the target multi-channel data in a relational database, including:

[0124] If the processing type of the target multi-channel data is a new creation type, storing the target multi-channel data in a relational database;

[0125] If the processing type of the target multi-channel data is a change type, querying the historical multi-channel data corresponding to the target multi-channel data from the relational database, and changing the historical multi-channel data based on the target multi-channel data;

[0126] If the creation time is within a long-term time period, the second processing module 440 processes the target multi-channel data in a non-relational database, including:

[0127] If the processing type of the target multi-channel data is a change type, historical multi-channel data corresponding to the target multi-channel data is queried from the non-relational database, and the historical multi-channel data is changed based on the target multi-channel data.

[0128] In an embodiment of the present application, the device further includes:

[0129] a detection module, configured to detect, at a preset frequency, historical multi-channel data within a preset time period with the earliest creation time in the relational database; wherein the preset time period is the time period that first falls within the long-term time period;

[0130] A storage module, configured to construct the historical multi-channel data into a combination of a master table and a detail table, convert the data into a preset format, and store the data in a temporary table; wherein the master table contains data of preset fields in the historical channel data; and the detail table contains other data in the historical channel data except the preset fields;

[0131] The separation module is used to separate the fields of the historical multi-channel data in the temporary table according to a preset separator and store the separated fields in a non-relational database.

[0132] In an embodiment of the present application, the device further includes:

[0133] The parsing module is used to obtain the user's query conditions through the front-end page, parse them to obtain the filter conditions, the result display page number, and the number of results displayed on a single page;

[0134] The return module is used to search in the relational database and / or non-relational database according to the screening conditions, the result display page number and the number of results displayed on a single page, and display the returned result multi-channel data on the front-end page.

[0135] In the embodiment of the present application, the return module searches the relational database and / or non-relational database according to the screening conditions, the result display page number, and the number of results displayed on a single page, including:

[0136] For the current page selected by the user for viewing, determine the target number of results to be displayed based on the total number of pages from the first page to the current page and the number of results displayed on a single page;

[0137] If the number of results obtained from the query of the relational database based on the screening condition is greater than or equal to the target number of results, the multi-channel data of the results obtained from the query of the relational database that meet the target number of results and are ranked first are displayed on the front-end page;

[0138] Otherwise, continue to query the remaining number of result multi-channel data from the non-relational database based on the screening conditions, and display the result multi-channel data that meets the target number of result items queried from the relational database and the non-relational database on the front-end page; wherein the remaining number is the difference between the target number of result items and the number of result items queried from the relational database.

[0139] In an embodiment of the present application, the device further includes:

[0140] A display module is used to display a list of master data in the multi-channel data obtained by querying the user's query conditions on the front-end page; wherein the list of master data is a list of data corresponding to a preset field in the query result;

[0141] The jump module is used to set a detail page jump link for each master data list, so that when the user clicks to view the detail page, the link will jump to the channel to which the master data list belongs for detailed data query.

[0142] A multi-channel data processing device provided in an embodiment of the present application can execute a multi-channel data processing method provided in any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method.

[0143] Figure 6 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0144] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0145] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless multi-channel data processing transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0146] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the multi-channel data processing method.

[0147] In some embodiments, the multi-channel data processing method can be implemented as a computer program that is tangibly contained in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the multi-channel data processing method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute the multi-channel data processing method in any other appropriate manner (e.g., by means of firmware).

[0148] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0149] Computer programs for implementing the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable multi-channel data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0150] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0151] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0152] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0153] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0154] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the multi-channel data processing method provided in any embodiment of the present application.

[0155] In the process of implementation, the computer program product can be written in one or more programming languages or a combination thereof to write computer program code for performing the operations of the present invention, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, through the Internet using an Internet service provider).

[0156] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired information of the technical solution of this application can be achieved. This document is not limited here.

[0157] The above specific embodiments do not constitute a limitation on the scope of protection of this application. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included within the scope of protection of this application.

Claims

1. A multi-channel data processing method, characterized in that: The method comprises: Continuously receive target multi-channel data from multiple channels through the application programming interface, and push each target multi-channel data to the message queue; Reading the target multi-channel data one by one from the message queue, and determining the creation time of the item corresponding to the target multi-channel data; If the creation time is within a contiguous time period, processing the target multi-channel data in a relational database; wherein the contiguous time period is a time period before the current time and less than a preset time interval from the current time; and the relational database stores the multi-channel data of the project whose creation time is within the contiguous time period; If the creation time is within a distant time period, the target multi-channel data is processed in a non-relational database; wherein the distant time period is a time period before the adjacent time period, and the non-relational database stores multi-channel data of projects whose creation time is within the distant time period.

2. The method according to claim 1, characterized in that Before continuously receiving target multi-channel data from multiple channels through the application programming interface, the method further includes: Collect user data from multiple channels and count each field in each user data; Aggregate all fields in all user data and create a unified data model that includes all fields; Accordingly, after continuously receiving target multi-channel data from multiple channels through the application programming interface, the method further includes: The value of each field in the target multi-channel data is filled into the corresponding field in the unified data model to form target multi-channel data that conforms to a unified format.

3. The method according to claim 1, characterized in that After reading the target multi-channel data one by one from the message queue, the method further includes: determining a processing type of the target multi-channel data according to a type of an application programming interface for receiving the target multi-channel data; wherein the processing type includes a new creation type and a change type; If the creation time is within a contiguous time period, the target multi-channel data is processed in a relational database, including: If the processing type of the target multi-channel data is a new creation type, storing the target multi-channel data in a relational database; If the processing type of the target multi-channel data is a change type, querying the historical multi-channel data corresponding to the target multi-channel data from the relational database, and changing the historical multi-channel data based on the target multi-channel data; If the creation time is in the distant time period, the target multi-channel data is processed in a non-relational database, including: If the processing type of the target multi-channel data is a change type, historical multi-channel data corresponding to the target multi-channel data is queried from the non-relational database, and the historical multi-channel data is changed based on the target multi-channel data.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Detecting historical multi-channel data within a preset time period with the earliest creation time in the relational database at a preset frequency; wherein the preset time period is the time period that first falls within the long-term time period; The historical multi-channel data is constructed into a combination of a master table and a detail table, and converted into a preset format and stored in a temporary table; wherein the master table contains data of preset fields in the historical channel data; and the detail table contains other data in the historical channel data except the preset fields; The historical multi-channel data in the temporary table is separated into various fields according to a preset separator and stored in a non-relational database.

5. The method according to claim 1, wherein The method further comprises: Obtain the user's query conditions through the front-end page, parse them to obtain the filter conditions, result display page number, and the number of results displayed on a single page; According to the filtering conditions, the result display page number and the number of results displayed on a single page, a search is performed in the relational database and / or non-relational database, and the returned result multi-channel data is displayed on the front-end page.

6. The method according to claim 5, characterized in that Searching the relational database and / or non-relational database according to the screening conditions, the result display page number, and the number of result items displayed on a single page includes: For the current page selected by the user for viewing, determine the target number of results to be displayed based on the total number of pages from the first page to the current page and the number of results displayed on a single page; If the number of results obtained from the query of the relational database based on the screening condition is greater than or equal to the target number of results, the multi-channel data of the results obtained from the query of the relational database that meet the target number of results and are ranked first are displayed on the front-end page; Otherwise, continue to query the remaining number of result multi-channel data from the non-relational database based on the screening conditions, and display the result multi-channel data that meets the target number of result items queried from the relational database and the non-relational database on the front-end page; wherein the remaining number is the difference between the target number of result items and the number of result items queried from the relational database.

7. The method according to claim 1, characterized in that The method further comprises: Displaying a list of master data in the multi-channel data obtained by querying the user's query conditions on the front-end page; wherein the master data list is a list of data corresponding to the preset fields in the query results; A detail page jump link is set for each master data list so that when the user clicks to view the detail page, the link will jump to the channel to which the master data list belongs for detailed data query.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the multi-channel data processing method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the multi-channel data processing method according to any one of claims 1 to 7 when executed.

10. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the multi-channel data processing method according to any one of claims 1 to 7.

Images