Unified data distribution method and system based on scene

Abstract

The invention discloses a scene-based unified data distribution method and system, and the method comprises the steps: receiving specific scene data sent by an upstream system, and waiting for a consumer to process the specific scene data; setting distribution rules of different scenes, and configuring one or more distribution rules in the same scene to generate a plurality of to-do tasks; generating corresponding to-be-handled tasks according to the distribution rule, and pushing the to-be-handled tasks in batches at regular time; converting the format of the specific scene data and the variable naming format according to the requirements of the target service; and the converted data is accurately sent to the target service, and flow limiting is realized in the distribution process so as to limit the data flow and the concurrent processing quantity. Through unified data distribution, the work of an upstream system in the aspects of data distribution and processing is greatly simplified, time and resources needed in the development process are reduced, and the cost of long-term maintenance of the system is reduced. An upstream system can operate more efficiently, and meanwhile the burden of developers is relieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of information processing, and relates to a unified data distribution method and system based on scenarios. Background Art

[0002] In the current rapidly developing field of information technology, data distribution and processing have become the core links in many business systems. The efficient operation of this link is directly related to the performance, stability, and business efficiency of the system. However, the existing data distribution solutions have exposed various deficiencies in practice, restricting the further optimization and expansion of business systems.

[0003] First of all, traditional data distribution solutions often require complex docking and data transmission between upstream systems and multiple downstream targets. This "one-to-many" data transmission mode not only increases the development difficulty and cost, but also, due to the complexity of the transmission link, is extremely prone to errors and delays in the data transmission process. This not only affects the timeliness of business processing but also increases the complexity of system maintenance.

[0004] Secondly, the existing solutions lack sufficient flexibility in dealing with different data distribution scenarios. Different business scenarios may require different distribution rules and task types, but the existing data distribution solutions often can only provide limited configuration options and cannot meet diverse needs. This limitation restricts the innovation ability and adaptability of business systems, making the system appear powerless when facing complex and changeable business environments.

[0005] In addition, the existing solutions also have deficiencies in supporting data structures and variable naming formats. With the continuous development of business systems and the rapid growth of data volume, the diversification of data structures and variable naming formats has become an inevitable trend. However, the existing data distribution solutions often can only support limited data structures and variable naming formats, resulting in additional conversion and processing of data during the distribution process, increasing the complexity and operating cost of the system.

[0006] In terms of exception handling, many existing data distribution systems either do not support a retry mechanism or have an imperfect retry mechanism design. During data transmission, transmission failures caused by network failures, system exceptions, etc. are inevitable. The lack of a perfect retry mechanism means that the system cannot perform effective self-recovery when facing such problems, thus increasing the risk of data transmission failure.

[0007] At the same time, the lack of a flow-limiting function is also a major defect of the existing data distribution solutions. In high-concurrency scenarios, the system is often prone to performance bottlenecks due to excessive request volumes, resulting in increased data transmission delays or even system crashes. However, the existing data distribution solutions often lack effective flow-limiting measures to deal with this situation, thus restricting the scalability and stability of the system.

[0008] Finally, the existing solutions also have deficiencies in abnormal situation warning. When abnormal situations occur during the data distribution process, if the system fails to issue warnings in a timely manner and take corresponding measures for handling, these problems may further deteriorate and affect the stability and reliability of the entire system. Existing data distribution solutions often lack effective warning mechanisms to detect and solve these problems in a timely manner.

[0009] In summary, the existing data distribution solutions have deficiencies in aspects such as docking complexity, flexibility, support for data structure and variable naming format, exception handling, flow limiting function, and abnormal situation warning. Therefore, it is necessary to develop a more efficient, flexible, and reliable data distribution solution to meet the growing needs of business systems. Summary of the Invention

[0010] The purpose of the present invention is to solve the problem that in the prior art, the upstream system may need to be docked with multiple downstream systems separately to send data, which leads to an increase in the complexity and error rate of the docking process, and to provide a scenario-based unified data distribution method and system.

[0011] To achieve the above object, the present invention adopts the following technical solutions: A scenario-based unified data distribution method includes the following steps: Receiving specific scenario data sent by an upstream system, where the specific scenario data includes at least various events and message sending events in the pre-loan, during-loan, and post-loan processing processes, and waiting for a consumer to process the specific scenario data; Setting distribution rules for different scenarios, and configuring one or more distribution rules for the same scenario to generate multiple to-do tasks; Generating corresponding to-do tasks according to the distribution rules, and regularly pushing the to-do tasks in batches; Converting the format and variable naming format of the specific scenario data according to the requirements of the target service; Accurately sending the data after conversion processing to the target service, and implementing flow limiting during the distribution process to limit the data flow and the number of concurrent processes.

[0012] The step of receiving specific scenario data sent by an upstream system, where the specific scenario data includes at least various events and message sending events in the pre-loan, during-loan, and post-loan processing processes, and waiting for a consumer to process the specific scenario data; specifically: Configuring a data receiving interface according to the requirements of the upstream system, where the interface includes a protocol, a port, and an address, pre-defining specific scenario data, and assigning a unique identifier to each scenario; Continuously monitor the configured interfaces to wait for data sent by the upstream system, verify the integrity and format of the received data to meet the expectations, and match the received data to predefined scenarios based on the data identifier or content; Store the received data in a temporary cache and wait for consumers to process it, and maintain the status of the processing progress for each scenario's data; When new data arrives, notify the registered consumers. When multiple consumers are interested in the same scenario, implement a load balancing strategy to ensure even distribution of data; Consumers retrieve data from the cache and process it. After the consumers finish processing the data, update the data status.

[0013] Configure one or more distribution rules for the same scenario to generate multiple to-do tasks. Specifically: Configure one or more distribution rules for each specific scenario. The distribution rules include pushing to Dabu Niu, big data platform, pushing notifications by channel, collection system, Feishu, real-time tags, SMS platform, and WeChat. The distribution rules define the operations to be performed or the types of notifications to be sent in this scenario; When the scenario is triggered, generate corresponding to-do tasks according to one or more distribution rules configured for this scenario; The execution of each distribution rule results in the generation of one or more to-do tasks to meet the specific requirements for operations or notifications in this scenario.

[0014] Generate corresponding to-do tasks according to the distribution rules and push the to-do tasks in batches at regular intervals. Specifically: Set up a scheduled task to specify the specific time and frequency for pushing to-do tasks in batches; Aggregate the to-do tasks before the scheduled task is executed, merge tasks of the same type or the same target to reduce the number of pushes and improve processing efficiency; When the scheduled task reaches the scheduled execution time, push the aggregated to-do tasks to the specified channels or system components according to the configured push rules; Collect push results and feedback information during the process of pushing tasks, record the information of tasks that fail to be pushed successfully, and at the same time try to re-push or generate an error report; Track the status of each to-do task, including generated, pushed, processed, and failed, and monitor the progress and effect of task processing.

[0015] Convert the format of the specific scenario data according to the requirements of the target service. Specifically: Parse the specific scenario data into an internal data structure. According to the data format requirements of the target service, convert the parsed internal data structure into the corresponding data format. When the target service requires the XML format, use an XML serialization tool or library to convert the internal data structure into an XML document that complies with the target service specification, where the tag names, attribute names, and text content of the XML document all comply with the target service specification; when the target service requires the JSON format, use a JSON serialization tool or library to convert the internal data structure into a JSON string that complies with the target service specification, where the key names, value types, and structures of the JSON object all comply with the target service specification.

[0016] The conversion of the naming format of the specific scenario variables is performed according to the requirements of the target service, specifically: According to the variable naming requirements of the target service, perform naming format conversion on the variable name. When the target service requires camel case naming, convert the variable name from underscore naming or other naming formats to camel case naming; if the target service requires underscore naming, convert the variable name from camel case naming or other naming formats to underscore naming; if the target service specifies a specific variable name, directly replace the variable name with the specified name; for variable values that need to be converted or formatted, including dates, times, and numbers, apply the corresponding conversion rules or formatting functions for processing.

[0017] The accurate sending of the data after conversion processing to the target service is specifically: Select the corresponding network transmission protocol according to the characteristics and requirements of the target service; Establish a network connection with the target service and configure network parameters, including at least the port number and timeout time; Package the data and send the data packet to the target service according to the defined transmission protocol and rules to complete data distribution and transmission.

[0018] The implementation of traffic limiting during the distribution process to limit the data traffic and the number of concurrent processes is specifically: Formulate a traffic limiting strategy according to the processing capacity and business requirements of the target service. The traffic limiting strategy is set based on one or more factors such as request rate, number of concurrent connections, and IP address; Implement the traffic limiting strategy using a traffic limiting algorithm. The traffic limiting algorithm includes one or more of the counter algorithm, leaky bucket algorithm, and token bucket algorithm. Implement the algorithm logic by writing code or using a traffic limiting framework; During the data transmission process, control the number of concurrent requests through semaphore and thread pool technologies, and dynamically adjust the number of concurrent requests according to the response time and processing capacity of the target service to optimize performance; Real-time monitor the performance metrics of data transmission and processing, including throughput, response time, and error rate, and dynamically adjust the flow-limiting strategy and concurrency control mechanism according to the monitoring results.

[0019] A scenario-based unified data distribution system, including the following modules: A data receiving module, used to receive specific scenario data sent by the upstream system. The specific scenario data covers various events and message sending events in the pre-loan, in-loan, and post-loan processing processes, and waits for consumers to process the specific scenario data; A distribution rule setting module, used to set distribution rules for different scenarios. One or more distribution rules can be configured for the same scenario to generate multiple to-do tasks; A to-do task generation and push module, which generates corresponding to-do tasks according to the distribution rules and has the function of periodically pushing the to-do tasks in batches; A data format conversion module, which converts the format and variable naming format of the specific scenario data according to the requirements of the target service to ensure data compatibility and accuracy; A data distribution and flow-limiting module, which accurately sends the data after conversion processing to the target service and implements a flow-limiting function during the distribution process to limit the data flow and the number of concurrent processes, thereby ensuring the stability and reliability of the system.

[0020] The system also includes an exception handling module and a warning module. The exception handling module retries when an exception occurs, and the number of retries is preset by the user; the warning module sends a warning message to relevant personnel in a timely manner when the exception handling module fails to succeed after retrying multiple times.

[0021] Compared with the prior art, the present invention has the following beneficial effects: Significantly reduce the working complexity and cost of the upstream system: Through the scenario-based unified data distribution method in the present invention, the work of the upstream system in data distribution and processing has been greatly simplified, not only reducing the time and resources required in the development process, but also reducing the long-term maintenance cost of the system. This enables the upstream system to operate more efficiently and at the same time reduces the burden on developers.

[0022] Enhance the flexibility of data adaptation and the application scope of the system: The present invention has excellent data adaptation capabilities, can meet the data requirements of different target services, and thus greatly expands the application scope of the system. In the face of various types or formats of data, it can provide effective adaptation solutions to ensure the accurate transmission and processing of data.

[0023] Ensure stable operation in complex system environments: Through integrated retry and rate-limiting functions, the system can maintain stable operation in complex and ever-changing network environments. The retry mechanism can automatically attempt to resend data when data transmission fails, while the rate-limiting function can effectively prevent system crashes or performance degradation caused by high-concurrency requests, reducing potential losses caused by exceptions and high concurrency.

[0024] Improve system reliability and user experience: The anomaly warning function can issue alerts in a timely manner when the system detects anomalies or potential problems, allowing operations and maintenance personnel to respond and resolve issues quickly. This timely warning mechanism greatly improves system reliability and optimizes the user experience, ensuring smoothness and satisfaction for users during use. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for use in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can be obtained based on these drawings.

[0026] Figure 1 It is a flowchart of the scenario-based unified data distribution method in the present invention; Figure 2 It is a schematic diagram of the specific working process of the scenario-based unified data distribution method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Usually, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0028] Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed present invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0029] It should be noted that similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0030] The present invention will be further described in detail below with reference to the accompanying drawings: Refer to Figure 1 , which is a flowchart of the scenario-based unified data distribution method in the present invention, including the following steps: S1. Receive specific scenario data sent by the upstream system. The specific scenario data includes at least various events and message sending events in the pre-loan, in-loan, and post-loan processing processes, and wait for consumers to process the specific scenario data. For example: pre-loan - public security real-name authentication, pre-loan - loan application successfully accepted, pre-loan - automatic approval passed, pre-loan - automatic approval rejected, in-loan - withdrawal application security check, post-loan - active repayment application process passed, Feishu - send message, email - send message, etc.

[0031] S1.1. Configure a data receiving interface according to the requirements of the upstream system. The interface includes a protocol, a port, and an address. Pre-define specific scenario data and assign a unique identifier to each scenario.

[0032] S1.2. Continuously monitor the configured interface to wait for data sent by the upstream system, verify whether the integrity and format of the received data meet the expectations, and match the received data to the pre-defined scenario according to the identifier or content of the data.

[0033] S1.3. Store the received data in a temporary cache and wait for consumers to process it. Maintain the status of the processing progress for each scenario data.

[0034] S1.4. When new data arrives, notify the registered consumers. When multiple consumers are interested in the same scenario, implement a load balancing strategy to ensure even distribution of data.

[0035] S1.5. Consumers obtain data from the cache and process it. After the consumers process the data, update the data status.

[0036] S2. Set distribution rules for different scenarios. Configure one or more distribution rules for the same scenario to generate multiple to-do tasks.

[0037] S2.1. Configure one or more distribution rules for each specific scenario. The distribution rules include pushing to Dabu Niu, the big data platform, pushing notifications by channel, the collection system, Feishu, real-time tags, the SMS platform, and WeChat. The distribution rules define the operations to be performed or the types of notifications to be sent in this scenario.

[0038] S2.2. When the scenario is triggered, generate corresponding to-do tasks according to one or more distribution rules configured for this scenario.

[0039] S2.3. The execution of each distribution rule results in the generation of one or more to-do tasks to meet the specific requirements for operations or notifications in this scenario.

[0040] Through this method, it can be ensured that in different scenarios, the system can flexibly and accurately generate corresponding to-do tasks according to the preset distribution rules, thereby improving the efficiency and accuracy of task processing.

[0041] S3. Generate corresponding to-do tasks according to the distribution rules and push the to-do tasks in batches at regular intervals.

[0042] S3.1. Set a scheduled task to specify the specific time and frequency for pushing to-do tasks in batches.

[0043] S3.2. Aggregate the to-do tasks before the scheduled task is executed, merge tasks of the same type or the same target, so as to reduce the number of pushes and improve the processing efficiency.

[0044] S3.3. When the scheduled task reaches the scheduled execution time, push the aggregated to-do tasks to the specified channel or system component according to the configured push rules.

[0045] S3.4. Collect push results and feedback information during the process of pushing tasks, record the information of tasks that fail to be pushed successfully, and at the same time try to re-push or generate an error report.

[0046] S3.5. Track the status of each to-do task, including generated, pushed, processed, and failed, and monitor the progress and effect of task processing.

[0047] S4. Convert the format of the specific scenario data and the variable naming format according to the requirements of the target service.

[0048] Parse the specific scenario data into an internal data structure, and according to the data format requirements of the target service, convert the parsed internal data structure into the corresponding data format. When the target service requires the XML format, use an XML serialization tool or library to convert the internal data structure into an XML document that conforms to the target service specification, where the tag names, attribute names, and text content of the XML document all conform to the target service specification; when the target service requires the JSON format, use a JSON serialization tool or library to convert the internal data structure into a JSON string that conforms to the target service specification, where the key names, value types, and structures of the JSON object all conform to the target service specification.

[0049] According to the variable naming requirements of the target service, perform naming format conversion on variable names. When the target service requires camel case naming, convert the variable name from underscore naming or other naming formats to camel case naming; if the target service requires underscore naming, convert the variable name from camel case naming or other naming formats to underscore naming; if the target service specifies a specific variable name, directly replace the variable name with the specified name; for variable values that need to be converted or formatted, including dates, times, and numbers, apply the corresponding conversion rules or formatting functions for processing.

[0050] S5, accurately send the data after conversion processing to the target service, and implement traffic limiting during the distribution process to limit the data traffic and the number of concurrent processes.

[0051] According to the characteristics and requirements of the target service, select the corresponding network transmission protocol; establish a network connection with the target service, configure network parameters, including at least the port number and timeout; pack the data, and send the data packet to the target service according to the defined transmission protocol and rules to complete data distribution and transmission.

[0052] According to the processing capacity and business requirements of the target service, formulate a traffic limiting strategy, and the traffic limiting strategy is set based on one or more factors such as request rate, number of concurrent connections, and IP address; Implement the traffic limiting strategy using a traffic limiting algorithm, and the traffic limiting algorithm includes one or more of the counter algorithm, leaky bucket algorithm, and token bucket algorithm, and implement the algorithm logic by writing code or using a traffic limiting framework; During data transmission, control the number of concurrent requests through semaphore and thread pool technologies, and dynamically adjust the number of concurrent requests according to the response time and processing capacity of the target service to optimize performance; Real-time monitor the performance metrics of data transmission and processing, including throughput, response time, and error rate, and dynamically adjust the traffic limiting strategy and concurrent control mechanism according to the monitoring results.

[0053] The scenario-based unified data distribution method in the present invention effectively solves many problems in the prior art. The upstream system only needs to send the data of a specific scenario to this service, greatly simplifying the docking process and reducing the error rate. The service will generate various to-do tasks according to the distribution rules bound to the scenario, such as pushing to partners, collection, App, SMS, WeChat, data warehouse, etc., meeting the requirements of different business scenarios. The to-do tasks can be accurately adapted according to the data structure and variable naming format required by the target service, supporting rich data structures such as xml and json, as well as variable patterns separated by camel case or underscores, enhancing the adaptability of the system. This solution supports exception retries, and the number of retries can be flexibly set. At the same time, it has a flow-limiting function, which can effectively handle high concurrency situations and ensure the stable performance of the system. In addition, it also supports exception warnings, notifying relevant personnel to handle exceptions in a timely manner and improving the reliability of the system.

[0054] An embodiment of the present invention is a scenario-based unified data distribution system, including the following modules: A data reception module, configured to receive specific scenario data sent by an upstream system, where the specific scenario data covers various events and message sending events in the pre-loan, in-loan, and post-loan processing processes, and waits for a consumer to process the specific scenario data; A distribution rule setting module, configured to set distribution rules for different scenarios, and one or more distribution rules can be configured for the same scenario to generate multiple to-do tasks; A to-do task generation and push module, which generates corresponding to-do tasks according to the distribution rules and has the function of periodically pushing the to-do tasks in batches; A data format conversion module, which converts the format and variable naming format of the specific scenario data according to the requirements of the target service to ensure data compatibility and accuracy; A data distribution and flow-limiting module, which accurately sends the data after conversion processing to the target service and implements a flow-limiting function during the distribution process to limit the data flow and the number of concurrent processes, thereby ensuring the stability and reliability of the system.

[0055] An exception handling module and a warning module, where the exception handling module retries when an exception occurs, and the number of retries is preset by the user; the warning module sends a warning message to relevant personnel in a timely manner when the exception handling module fails to succeed after retrying multiple times.

[0056] The present invention has significant advantages. First, it greatly simplifies the work of the upstream system and reduces the development and maintenance costs. Second, its flexible data adaptation ability meets the needs of various target services and expands the application scope of the system. The retry and rate-limiting functions ensure the stable operation of the system in complex environments and reduce losses caused by exceptions and high concurrency. The exception warning function enables problems to be handled in a timely manner, improving the reliability of the system and the user experience. In summary, the present invention effectively improves the efficiency, stability, and adaptability of data distribution. Embodiment

[0057] 1. Scenario definition, different scenarios need to be defined. Each scenario represents a specific business requirement or trigger condition. For example: Scenario A: User registration is successful Scenario B: User fails to repay the loan overdue Scenario C: User completes a transaction Scenario D: User birthday reminder 2. Distribution rule configuration In the rule configuration module, configure one or more distribution rules for each scenario. The distribution rules determine which operations or notifications the system needs to perform in that scenario. The distribution rules can include the following: Push to Dabiniu: Push the task to the Dabiniu system for processing.

[0058] Big data platform: Push relevant data to the big data platform for analysis or storage.

[0059] Push notifications by channel: Push notifications to different channels (such as APP, email, etc.) according to user preferences or channel settings.

[0060] Collection system: If it involves an overdue or collection scenario, push the task to the collection system for processing.

[0061] Feishu: Send notifications or task reminders through Feishu.

[0062] Real-time tags: Update user tags in real time according to user behavior or status.

[0063] SMS platform: Send SMS notifications through the SMS platform.

[0064] WeChat: Send notifications through WeChat official accounts or mini-programs.

[0065] Example: Scenario A (User registration is successful): Distribution rule 1: Push to Dabiniu and generate a task for successful user registration.

[0066] Distribution rule 2: Push notifications by channel and send a welcome message to the user.

[0067] Distribution Rule 3: Real-time Tagging, updating the user tag to "New User".

[0068] Scenario B (User overdue on repayment): Distribution Rule 1: Push to Dabu Niu, generating an overdue handling task.

[0069] Distribution Rule 2: Debt collection system, pushing the overdue user information to the debt collection system.

[0070] Distribution Rule 3: SMS platform, sending overdue reminder SMS.

[0071] Distribution Rule 4: WeChat, sending overdue reminder notice.

[0072] 3. Task Generation When a certain scenario is triggered, the system will generate corresponding to-do tasks according to the distribution rules configured for that scenario. Each distribution rule may generate one or more tasks.

[0073] Example: Scenario A (User registration successful) is triggered: According to Distribution Rule 1, generate a task of "Push to Dabu Niu".

[0074] According to Distribution Rule 2, generate a task of "Push notification by channel".

[0075] According to Distribution Rule 3, generate a task of "Real-time tag update".

[0076] Scenario B (User overdue on repayment) is triggered: According to Distribution Rule 1, generate a task of "Push to Dabu Niu".

[0077] According to Distribution Rule 2, generate a task of "Debt collection system processing".

[0078] According to Distribution Rule 3, generate a task of "SMS platform sending SMS".

[0079] According to Distribution Rule 4, generate a task of "WeChat sending notification".

[0080] 4. Task Distribution and Execution The generated to-do tasks will be pushed to the corresponding systems or platforms for processing according to the distribution rules. Each system or platform will perform corresponding operations according to the content of the task.

[0081] Example: For Scenario A: After receiving the task, the Dabu Niu system processes the subsequent operations for successful user registration.

[0082] The task of pushing notifications by channel will select channels such as the APP and email to send welcome messages according to the user's preferences.

[0083] The real-time tagging system will update the user's tag to "new user".

[0084] For Scenario B: After receiving the task, the Dabuniu system processes the subsequent operations of overdue users.

[0085] The collection system will initiate the collection process according to the pushed task.

[0086] The SMS platform will send overdue reminder SMS to users.

[0087] The WeChat platform will send overdue reminder notifications to users.

[0088] 5. Task Status Tracking and Feedback During the execution of each task, the system will track the task status, such as "pending", "processing", "completed", etc., and provide feedback based on the task execution result. If the task execution fails, the system may retry according to the configured retry rules or trigger other processing flows.

[0089] 6. Expansion and Optimization of Scenarios and Rules As the business requirements change, scenarios and distribution rules can be added, modified, or deleted in the rule configuration module at any time. Corresponding tasks are generated according to the latest configuration to ensure the flexibility and scalability of the business logic.

[0090] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A scenario-based unified data distribution method, characterized in that: The following steps are involved: Receive specific scenario data sent by the upstream system, the specific scenario data at least including various events in the pre-loan, mid-loan and post-loan processing processes and message sending events, and wait for consumers to process the specific scenario data; Set distribution rules for different scenarios. Configure one or more distribution rules for the same scenario to generate multiple to-do tasks. Generate corresponding to-do tasks according to the distribution rules, and push the to-do tasks in batches at regular intervals; Convert the format of the specific scenario data and the variable naming format according to the requirements of the target service; The converted data is accurately sent to the target service, and current limiting is implemented during the distribution process to limit data traffic and the number of concurrent processing.

2. A scenario-based unified data distribution method as claimed in claim 1, characterized in that: The specific scenario data sent by the upstream system is received, and the specific scenario data at least includes various events in the pre-loan, mid-loan and post-loan processing processes and message sending events, and waits for the consumer to process the specific scenario data; specifically: Configure a data receiving interface according to the requirements of the upstream system, the interface including a protocol, port and address, predefine specific scenario data, and assign a unique identifier to each scenario; Continuously listen to the configured interface to wait for the upstream system to send data, verify whether the integrity and format of the received data are as expected, and match the data to the predefined scenario based on the identifier or content; Store the received data in a temporary cache, waiting for consumer processing, and maintain the status of the processing progress for each scene data; When new data arrives, registered consumers are notified. When multiple consumers are interested in the same scenario, a load balancing strategy is implemented to ensure even distribution of data. The consumer obtains data from the cache and processes it. After processing the data, the consumer updates the data status.

3. A scenario-based unified data distribution method as claimed in claim 1, characterized in that: The same scenario configures one or more distribution rules to generate multiple to-do tasks, specifically: One or more distribution rules are configured for each specific scenario. The distribution rules include push notifications to Dabuniu, big data platform, push notifications by channel, collection system, Feishu, real-time tags, SMS platform and WeChat. The distribution rules define the operations to be performed or the types of notifications to be sent in the scenario. When the scenario is triggered, a corresponding to-do task is generated according to one or more distribution rules configured for the scenario; The execution of each distribution rule results in the generation of one or more to-do tasks to meet the specific needs of the operation or notification in the scenario.

4. A scenario-based unified data distribution method as claimed in claim 1, characterized in that: The corresponding to-do tasks are generated according to the distribution rules, and the to-do tasks are pushed in batches at regular intervals; specifically: Set up scheduled tasks to specify the specific time and frequency of batch push of pending tasks; Aggregate pending tasks before scheduled tasks are executed, and merge tasks of the same type or with the same goal to reduce the number of pushes and improve processing efficiency; When the scheduled task reaches the scheduled execution time, the aggregated to-do tasks are pushed in batches to the specified channel or system component according to the configured push rules; Collect push results and feedback information during the task push process, record task information that failed to be pushed, and try to push again or generate error reports; Track the status of each to-do task, including generated, pushed, processed, and failed, and monitor the progress and effect of task processing.

5. The scenario-based unified data distribution method according to claim 1, characterized in that: The format of the specific scene data is converted according to the requirements of the target service, specifically: Parse the specific scenario data into an internal data structure, and convert the parsed internal data structure into the corresponding data format according to the data format requirements of the target service. When the target service requires XML format, use XML serialization tools or libraries to convert the internal data structure into an XML document that complies with the target service specifications, where the tag names, attribute names, and text content of the XML document all comply with the specifications of the target service; when the target service requires JSON format, use JSON serialization tools or libraries to convert the internal data structure into a JSON string that complies with the target service specifications, where the key name, value type, and structure of the JSON object all comply with the specifications of the target service.

6. A scenario-based unified data distribution method as claimed in claim 1, characterized in that: The specific scene variable naming format is converted according to the requirements of the target service, specifically: According to the variable naming requirements of the target service, perform naming format conversion on the variable name. When the target service requires camel case naming, convert the variable name from underscore naming or other naming formats to camel case naming; if the target service requires underscore naming, convert the variable name from camel case naming or other naming formats to underscore naming; if the target service specifies a specific variable name, directly replace the variable name with the specified name; for variable values ​​that need to be converted or formatted, including dates, times, and numbers, apply corresponding conversion rules or formatting functions for processing.

7. A scenario-based unified data distribution method as claimed in claim 1, characterized in that: The method of accurately sending the converted data to the target service is as follows: Select the corresponding network transmission protocol according to the characteristics and requirements of the target service; Establish a network connection with the target service and configure network parameters, including at least the port number and timeout period; Pack the data and send the data packets to the target service according to the defined transmission protocols and rules to complete data distribution and transmission.

8. A scenario-based unified data distribution method as claimed in claim 1, characterized in that: The current limiting is implemented during the distribution process to limit the data flow and the number of concurrent processing, specifically: According to the processing capacity and business needs of the target service, a flow limiting strategy is formulated, and the flow limiting strategy is set based on one or more factors including request rate, number of concurrent connections, and IP address; A current limiting algorithm is used to implement a current limiting strategy. The current limiting algorithm includes one or more of a counter algorithm, a funnel bucket algorithm, and a token bucket algorithm. The algorithm logic is implemented by writing code or using a current limiting framework. During data transmission, the number of concurrent requests is controlled through semaphore and thread pool technology, and the number of concurrent requests is dynamically adjusted according to the response time and processing capacity of the target service to optimize performance; Monitor the performance indicators of data transmission and processing in real time, including throughput, response time, and error rate, and dynamically adjust the current limiting strategy and concurrency control mechanism based on the monitoring results.

9. A scenario-based unified data distribution system, characterized in that: Includes the following modules: A data receiving module is used to receive specific scenario data sent by an upstream system, wherein the specific scenario data covers various events and message sending events in the pre-loan, mid-loan and post-loan processing processes, and wait for consumers to process the specific scenario data; The distribution rule setting module is used to set the distribution rules in different scenarios. One or more distribution rules can be configured in the same scenario to generate multiple to-do tasks. The to-do task generation and push module generates corresponding to-do tasks according to the distribution rules and has the function of regularly pushing the to-do tasks in batches; The data format conversion module converts the format of the specific scenario data and the variable naming format according to the requirements of the target service to ensure the compatibility and accuracy of the data; The data distribution and current limiting module accurately sends the converted data to the target service and implements the current limiting function during the distribution process to limit the data flow and the number of concurrent processing, thereby ensuring the stability and reliability of the system.

10. A scenario-based unified data distribution system as claimed in claim 9, characterized in that: The system also includes an exception handling module and an early warning module. The exception handling module retries when encountering an exception, and the number of retries is preset by the user; the early warning module sends early warning information to relevant personnel in a timely manner when the exception handling module retries multiple times and still fails.

Images