Business data processing method, apparatus, device, and storage medium

Abstract

The application relates to the technical field of data processing, in particular to a business data processing method and device, equipment and a storage medium. The method comprises the following steps: obtaining a to-be-processed data source, the to-be-processed data source comprising user information and business data; performing association processing on the business data and a data lake table based on the user information to obtain a first association result, the data lake table being used for indicating a table of label information corresponding to the business data; performing screening processing on the first association result according to a rule broadcast stream to obtain a target association result, the rule broadcast stream being used for indicating a screening rule corresponding to the business data; and determining corresponding target business data according to the target association result and performing push processing on the target business data. The method realizes real-time computing and processing of data and push processing, improves the efficiency and accuracy of real-time data processing, realizes accurate push processing of data, and provides real-time, efficient and accurate data processing services for users.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a business data processing method, apparatus, device, and storage medium. Background Technology

[0002] Real-time data processing refers to the process by which a system can instantly receive, process, and analyze data streams and respond quickly. It emphasizes the real-time nature of data processing, requiring data to be processed immediately after its generation in order to obtain valuable information promptly.

[0003] Currently, the industry's focus on real-time data processing mainly involves orchestrating multi-source heterogeneous data.

[0004] However, the existing technologies mentioned above lack processing solutions for mobile number portability data in the telecommunications industry, particularly for real-time calculation and processing of data through combined condition filtering. Therefore, the lack of such processing solutions for mobile number portability data in the telecommunications industry, especially for real-time calculation and processing of data through combined condition filtering, is an urgent issue that needs to be addressed. Summary of the Invention

[0005] This application provides a business data processing method, apparatus, device, and storage medium to solve the aforementioned problems in the prior art, namely, the lack of processing solutions for number portability data in the telecommunications industry, especially for real-time calculation and processing of data through combined condition filtering.

[0006] Firstly, this application provides a business data processing method, the method comprising:

[0007] Obtain the data source to be processed, which includes user information and business data. Based on the user information, the business data is associated with the data lake table to obtain a first association result. The data lake table is used to indicate the table corresponding to the tag information of the business data.

[0008] The first association result is filtered according to the rule broadcast stream to obtain the target association result. The rule broadcast stream is used to indicate the filtering rules corresponding to the business data.

[0009] Based on the target association results, the corresponding target business data is determined, and the target business data is pushed to the system.

[0010] Optionally, before obtaining the data source to be processed, the method further includes:

[0011] Based on the user information obtained from the data source in real time, the data source is stored in the memory cache to obtain the storage result;

[0012] The processing operator based on the stream processing framework performs parallel reading processing on the stored results to obtain the data source to be processed after parallel reading processing. The processing operator includes multiple degrees of parallelism, and the parallel reading processing is performed according to the multiple degrees of parallelism. Each degree of parallelism contains corresponding unique identification information.

[0013] Optionally, the step of associating the business data with the data lake table based on the user information to obtain a first association result includes:

[0014] A corresponding data lake table is established based on the data source to be processed, and the data lake table includes: user business tag information;

[0015] The data source to be processed is associated with the data lake table based on the user information to obtain a first association result. The first association result includes the user business tag information and corresponds to the rule information of the rule broadcast stream for filtering.

[0016] Optionally, before filtering the first association result according to the rule broadcast stream to obtain the target association result, the method further includes:

[0017] The full set of rule data is determined based on the data source to be processed. Each rule in the full set of rule data corresponds to the rule information of the data source to be processed. The rule information includes user business rule information.

[0018] Optionally, the step of filtering the first association result according to the rule broadcast stream to obtain the target association result includes:

[0019] Determine the first rule broadcast stream corresponding to the first association result;

[0020] The first association result is filtered according to the first rule broadcast stream to obtain the target association result.

[0021] Optionally, the step of filtering the first association result according to the first rule broadcast stream to obtain the target association result includes:

[0022] Determine the user service rule information corresponding to the first rule broadcast stream;

[0023] Based on the user business rule information corresponding to the first rule broadcast stream, the user business tag information corresponding to the first association result is filtered to obtain the target association result.

[0024] Optionally, the process of pushing the target business data includes:

[0025] The target business data is stored in the stream processing platform so that the stream processing platform can push and process the target business data in real time.

[0026] Secondly, this application provides a business data processing apparatus, the apparatus comprising:

[0027] The acquisition module is used to acquire the data source to be processed, which includes user information and business data, and to perform association processing between the business data and the data lake table based on the user information to obtain a first association result. The data lake table is used to indicate the table corresponding to the tag information of the business data.

[0028] The processing module is used to filter the first association result according to the rule broadcast stream to obtain the target association result, wherein the rule broadcast stream is used to indicate the filtering rules corresponding to the business data;

[0029] The push module is used to determine the corresponding target business data based on the target association result, and to push the target business data.

[0030] Optionally, the processing module is further configured to store the data source into a memory cache based on the user information obtained in real time, and obtain the storage result;

[0031] The processing module is further configured to perform parallel reading processing on the stored results based on the processing operator of the stream processing framework, to obtain the data source to be processed after parallel reading processing. The processing operator includes multiple degrees of parallelism, and the parallel reading processing is performed according to the multiple degrees of parallelism. Each degree of parallelism contains corresponding unique identification information.

[0032] Optionally, the processing module is further configured to establish a corresponding data lake table based on the data source to be processed, wherein the data lake table includes: user business tag information;

[0033] The processing module is further configured to associate the data source to be processed with the data lake table according to the user information to obtain a first association result. The first association result includes the user business tag information and corresponds to the rule information of the rule broadcast stream for filtering.

[0034] Optionally, the device further includes: a determining module;

[0035] The determining module is used to determine full rule data based on the data source to be processed. Each rule data in the full rule data corresponds to the rule information of the data source to be processed. The rule information includes user business rule information.

[0036] Optionally, the determining module is further configured to determine the first rule broadcast stream corresponding to the first association result;

[0037] The processing module is further configured to filter the first association result according to the first rule broadcast stream to obtain the target association result.

[0038] Optionally, the determining module is further configured to determine the user service rule information corresponding to the first rule broadcast stream;

[0039] The processing module is further configured to filter the user business tag information corresponding to the first association result based on the user business rule information corresponding to the first rule broadcast stream, so as to obtain the target association result.

[0040] Optionally, the push module is further configured to store the target business data into a stream processing platform so that the stream processing platform can perform real-time push processing on the target business data.

[0041] Thirdly, this application provides a business data processing device, including: at least one processor and a memory;

[0042] The memory stores computer-executed instructions;

[0043] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the business data processing method as described in the first aspect and various possible implementations of the first aspect above.

[0044] Fourthly, embodiments of the present invention provide a readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the business data processing method described in the first aspect and various possible implementations of the first aspect.

[0045] This application provides a business data processing method, apparatus, device, and storage medium. The method acquires a data source to be processed, which includes user information and business data. Based on the user information, it associates the business data with a data lake table to obtain a first association result. The data lake table indicates the table corresponding to the tag information of the business data. The first association result is then filtered according to a rule broadcast stream to obtain a target association result. The rule broadcast stream indicates the filtering rules corresponding to the business data. Based on the target association result, the corresponding target business data is determined and pushed to the target business data. This method filters the data lake table containing tags of the data source to be processed using the filtering rules of the rule broadcast stream and pushes the filtering results in real time. This achieves real-time data calculation and processing and push, improving the efficiency and accuracy of real-time data processing and providing users with real-time, efficient, and accurate data processing services. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0047] Figure 1 This application provides a process for a business data processing method. Figure 1 ;

[0048] Figure 2 This application provides a process for a business data processing method. Figure 2 ;

[0049] Figure 3 This is a schematic diagram of the structure of a business data processing device provided in this application;

[0050] Figure 4 This is a schematic diagram of the structure of a business data processing device provided in this application.

[0051] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein.

[0054] In this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0055] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0056] First, let me explain the terms used in this application:

[0057] Data Lake: A data lake is a repository or system that stores data in its raw format, capable of storing large amounts of structured and unstructured data. It allows data to be stored in its original state without prior structuring, thus preserving the integrity and originality of the data.

[0058] Data Lake Tables: In a data lake environment, the concept of tables, or data lake tables, is introduced to better organize and access data. Data lake tables can be physical tables or virtual tables, each with different characteristics and uses. As a key component of the data lake architecture, a data lake table is an abstraction layer above the data lake file format. It allows users to encapsulate large amounts of data from a distributed file system into table structures similar to those in traditional databases, facilitating querying and analysis. These tables not only support complex query logic and data transformations but also provide efficient data access and processing capabilities while maintaining data originality and flexibility. Data lake tables can include both physical and virtual tables. Physical tables directly relate to the actual stored data, while virtual tables are more like definitions or views, describing how to query data from one or more physical tables. This design enables data engineers and analysts to create complex data processing and query workflows to meet diverse business needs.

[0059] Stream processing frameworks are system architectures used to process real-time data streams. They allow applications to receive and process data in the form of streams, which are typically generated continuously and in real time without explicit end boundaries. Stream processing frameworks are widely used in real-time data analysis, monitoring, machine learning, and other fields. Apache Flink (or simply Flink) is an open-source stream processing framework for handling unbounded and bounded data streams. It provides powerful window operations, state management, and fault tolerance mechanisms, supports event-time and processing-time, and is a top-level Apache project designed to process unbounded and bounded data streams in a high-throughput, low-latency, and highly available manner. It supports in-memory computation and can run in all common cluster environments. Flink supports not only stream processing but also batch processing, achieving integrated batch and stream processing, while providing robust state management and fault tolerance mechanisms to ensure the accuracy and consistency of data processing. Flink's data interfaces include data stream interfaces, dataset interfaces, and table interfaces, supporting multiple programming languages, allowing developers to flexibly build complex data processing applications. Flink has been widely used globally in scenarios such as real-time intelligent recommendation, complex event processing, real-time fraud detection, and real-time data warehousing, becoming an important tool in the field of real-time big data processing.

[0060] Memory cache: A memory cache is a mechanism that uses computer memory to store temporary data. It improves data access speed and system performance by storing frequently accessed data or calculation results in memory, reducing access to slower storage media (such as hard drives or remote databases). Memory caches are widely used in various software systems and database management systems.

[0061] Parallelism: In stream processing frameworks, parallelism refers to the number of parallel tasks per operator. An operator can be divided into multiple parallel subtasks, which can be executed simultaneously to improve overall processing performance. Each subtask is executed in an independent task slot, and a task slot corresponds to a computing resource (such as a thread or a process).

[0062] Rule broadcast stream: A rule broadcast stream refers to broadcasting a data stream (rule stream) containing global rules or configurations to all tasks processing other data streams during stream processing. This allows each task to obtain the latest rules or configurations in real time and process its received data streams accordingly.

[0063] Data analytics engines are software systems or toolsets specifically designed for processing, analyzing, and mining data. They integrate various algorithms and technologies to quickly and efficiently perform data cleaning, transformation, aggregation, visualization, and other operations, helping users extract valuable information and insights from massive amounts of data. Data analytics engines support complex data queries, statistical analysis, machine learning model training and prediction, providing powerful data support for enterprise decision-making, scientific research, product development, and other fields.

[0064] Currently, the industry's real-time data processing mainly focuses on orchestrating multi-source heterogeneous data.

[0065] However, the industry lacks solutions for processing mobile number portability data, especially for real-time calculation and processing of data through combined condition filtering.

[0066] To address the aforementioned issues, this application provides a business data processing method. This method first acquires the data source to be processed, then associates the business data and corresponding tag information of the data source with a data lake based on user information within the data source. Next, it filters the association results according to the filtering rules corresponding to the business data in the rule broadcast stream, obtaining the target association result. Finally, it pushes the target business data corresponding to the association result. This method filters the data lake containing tags of the data source to be processed through the filtering rules of the rule broadcast stream and pushes the filtering results in real time, achieving real-time data calculation, processing, and push. It improves the efficiency and accuracy of real-time data processing and provides users with real-time, efficient, and accurate data processing services.

[0067] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0068] Figure 1 This is a flowchart illustrating a business data processing method provided in an embodiment of this application. Figure 1 .like Figure 1 As shown in this embodiment, the business data processing method includes:

[0069] S101: Obtain the data source to be processed, which includes user information and business data, and perform association processing between the business data and the data lake table based on the user information to obtain the first association result.

[0070] The data source to be processed can be, for example, a number portability service database. The user information corresponding to this database may include, for example, user number information. The service data may include, for example, user gender information, user age information, user group information, and user location field information. The data lake table is used to indicate the tag information corresponding to the service data. This tag information may include, for example, user group information, user age information, user gender information, user number portability time information, user's porting-in and porting-out network information, user's original network information, and user location field information. The first association result may be, for example, the association result of all service data including the tag information corresponding to the service data.

[0071] The data lake table can be, for example, Apache Paimon (Paimon for short). It can also be any existing data lake table that has the same function as Paimon and can achieve the effect of this embodiment. No specific limitation is made here.

[0072] Understandably, before acquiring the data source to be processed, the real-time acquired data source can be stored in a memory cache. Then, based on the processing operators of the stream processing framework, the stored data source can be read and processed in parallel. This can improve the system's real-time processing speed of the data source, decompose the stored data source into multiple parallel execution parts, significantly improve the processing speed and system throughput, and reduce the system's real-time processing time.

[0073] S102: The first association result is filtered according to the rule broadcast stream to obtain the target association result.

[0074] The rule broadcast stream is used to indicate the filtering rules corresponding to the business data. The filtering process can be, for example, based on the filtering rules in the rule broadcast stream, filtering the association results of all business data, including the tag information corresponding to the business data. The target association result can include, for example, the remaining association result information after the first association result has been filtered by the filtering rules in the rule broadcast stream.

[0075] Understandably, before proceeding with this step, one could determine the filtering rules included in the rule broadcast stream by identifying the full set of rule data from the data source to be processed.

[0076] S103: Based on the target association result, determine the corresponding target business data and push the target business data.

[0077] One method for pushing target business data is to store the target business data in a stream processing platform and then use that platform to perform real-time push processing of the target business data.

[0078] Understandably, this step is based on the stream processing platform to complete the real-time push processing of target business data, thereby providing users with real-time, efficient and accurate data processing services.

[0079] The business data processing method provided in this embodiment first obtains the data source to be processed, then associates the business data in the data source with a data lake table containing the corresponding tag information of the business data based on the user information in the data source. Next, it filters the association results according to the filtering rules corresponding to the business data in the rule broadcast stream to obtain the target association result. Finally, it pushes the target business data corresponding to the association result. This method filters the data lake table containing the tags of the data source to be processed through the filtering rules of the rule broadcast stream and pushes the filtering results in real time, realizing real-time data calculation and processing and pushing. It improves the efficiency and accuracy of real-time data processing and achieves precise data push, providing users with real-time, efficient, and accurate data processing services.

[0080] Figure 2 A flowchart illustrating a business data processing method provided in this application embodiment. Figure 2 .like Figure 2 As shown, this embodiment is... Figure 1 Based on the embodiments, the business data processing method is described in detail below. The business data processing method shown in this embodiment includes:

[0081] S201: Based on the user information obtained from the data source in real time, the data source is stored in the memory cache to obtain the storage result.

[0082] The data source can be, for example, number portability service data information, which may include historical full number portability service data information and real-time incremental number portability service data information. The user information may include user number information. The memory cache may be, for example, a Redis (Remote Dictionary Server) cache, which may be any existing memory cache with the same function as the Redis cache, without any specific restrictions.

[0083] Understandably, this step stores the data source in a memory cache. For example, user group information corresponding to the user information can also be stored in the memory cache. The purpose of storing the data source in the memory cache is to leverage the high read and access speed of the memory cache to improve the read and access speed of the data source, thereby accelerating the system's response speed and performance, and improving the system's real-time processing speed of the data source.

[0084] S202: A processing operator based on a stream processing framework performs parallel reading processing on the stored results to obtain the data source to be processed after parallel reading processing.

[0085] The stream processing framework can be, for example, Flink. This framework can be any existing technology with similar functionality to Flink, without specific limitations. The processing operators can include multiple degrees of parallelism, and parallel read processing is performed based on these multiple degrees of parallelism. Each degree of parallelism contains a unique identifier to ensure data consistency and correctness during parallel read processing. This stream processing framework can, for example, perform dynamic detection processing on the data source to be processed.

[0086] Understandably, this step, based on the processing operators of the stream processing framework, aims to significantly improve the speed and efficiency of data processing by reading and processing the stored results in parallel. At the same time, it enhances the scalability and fault tolerance of the system. By reading and processing data in parallel, multiple processing units can work simultaneously, reducing the overall processing time, achieving effective resource utilization and load balancing, thereby improving the system's speed of real-time data processing.

[0087] S203: Establish a corresponding data lake table based on the data source to be processed.

[0088] The data lake table includes user business tag information, which may include, for example, user group information, user age information, user gender information, user number portability time information, user porting in and out network information, user original network information, and user home location field information.

[0089] Understandably, the purpose of this step is to perform subsequent filtering processing on the results of associating the data lake table containing user business tag information with the data source to be processed.

[0090] S204: Based on the user information, perform association processing between the data source to be processed and the data lake table to obtain the first association result.

[0091] The user information may include, for example, user number information. The association processing may include, for example, as follows: based on the user number information, associating the user business tag information of the data source to be processed and the user business tag information in the data lake table to form a first association result of the full amount of business data including the tag information corresponding to the business data. The first association result corresponds to the rule information of the rule broadcast stream that has been filtered.

[0092] Understandably, the purpose of this step, which associates the data source to be processed with the data lake table based on user information, is to enable the filtering and processing of all business data, including the tag information corresponding to the business data, based on the characteristics of the data lake table. Through association with the data lake table, high throughput, low latency data ingestion, and real-time query capabilities can be provided, thereby improving the speed of real-time data processing in the system.

[0093] S205: Determine the full set of rule data based on the data source to be processed. Each rule data in the full set of rule data corresponds to the rule information of the data source to be processed. The rule information includes: user business rule information.

[0094] The full set of rule data can be created through a development interface. This rule data can include, for example, rule name information, rule filtering conditions (user age information, user number portability time information, user porting in and out network information), rule start time information, and rule deactivation time information. This rule information is written into a rule table for subsequent reading. This rule table can be a database table, and different rule information corresponds to different parameter conditions and database table data in the development interface.

[0095] Understandably, this step determines the full set of rule data based on the characteristics of the data source to be processed. Each rule in this set corresponds to the relevant rule information in the data source, covering all aspects of the user's business rule information, thus ensuring the accuracy and efficiency of the data processing process.

[0096] S206: Determine the first rule broadcast stream corresponding to the first association result.

[0097] The first rule broadcast stream can be, for example, a rule broadcast stream that has the same tag information as the tag information corresponding to the business data in the first association result.

[0098] S207: Determine the user service rule information corresponding to the first rule broadcast stream.

[0099] The user business rule information corresponding to the first rule broadcast stream can, for example, flexibly combine the filtering conditions of the rules in the user business rule information according to actual needs to achieve rule filtering under different business requirements. This flexible combination method can be, for example, relational expression concatenation, which can be, for example, a dynamic expression evaluation engine in existing technology, such as AviatorEvaluator. The flexible combination of the rule's filtering conditions can, for example, include, when performing filtering conditions for number portability user profiles, user profile filtering fields such as user gender information, user age information, and user location information.

[0100] S208: Based on the user business rule information corresponding to the first rule broadcast stream, filter the user business tag information corresponding to the first association result to obtain the target association result.

[0101] Understandably, this step can be executed, for example, based on the processing operators of the stream processing framework. After completing the filtering and processing of user business tag information, the target association results of the filtering and processing can be pushed to the stream processing platform and the data analysis engine to provide subsequent real-time data processing services.

[0102] The filtering process for the user business tag information corresponding to the first association result can be, for example, in the process of processing user profile business data for number portability, the filtering condition information in the rule name information of the first rule broadcast stream rule information corresponding to the user profile business data can include, for example, user gender information, user age information, and user location field information. The user business tag information of the corresponding first association result can include, for example, user group information, user age information, user gender information, user number portability time information, user porting in and out network information, user original network information, and user location field information. This filtering process can be implemented, for example, through simple fuzzy matching.

[0103] This step then pushes the data to the stream processing platform using fields from the data analysis engine, such as user number information, user number portability time information, user location field information, and the time information of pushing to the stream processing platform.

[0104] S209: Based on the target association result, determine the corresponding target business data and store the target business data in the stream processing platform so that the stream processing platform can push and process the target business data in real time.

[0105] The stream processing platform can be, for example, the Apache Kafka (Kafka for short) stream processing platform. It can also be any existing platform with the same functionality as Kafka, without any specific restrictions.

[0106] Understandably, the purpose of this step, which involves real-time push processing of target business data by the stream processing platform, is to achieve high throughput processing of large amounts of target business data. At the same time, it aims to achieve high scalability, persistence, and fault tolerance in real-time data processing based on the stream processing platform, thereby improving the user's experience in real-time data processing.

[0107] In some possible embodiments, while storing the target business data into the stream processing platform, the target business data can also be stored into the data analysis engine, so that the data analysis engine can perform real-time storage processing on the target business data.

[0108] The data analysis engine can be, for example, Elasticsearch (ES), or any existing analysis engine with the same functionality as ES, without any specific restrictions.

[0109] Understandably, the purpose of this step, which uses a data analysis engine to store and process target business data in real time, is to provide powerful full-text search capabilities, complex query and aggregation operations when users are processing data in real time. This ensures the reliability of the data and the stability of the system, thereby providing a powerful and efficient data analysis engine for real-time data processing.

[0110] It is worth mentioning that the embodiments of this application provide a real-time processing process for business data information corresponding to a rule broadcast stream. In actual use, the business data information will be simultaneously filtered and processed based on multiple rule broadcast streams, thereby providing a real-time processing system for multiple business data information. The multiple rule broadcast streams may include, for example, a second rule broadcast stream, a third rule broadcast stream, and a fourth rule broadcast stream.

[0111] The business data processing method provided in this embodiment improves the reading and access speed of the data source by storing the real-time acquired data source in a memory cache, thereby accelerating the system's response speed and performance and improving the system's real-time processing speed of the data source. Based on the processing operator of the stream processing framework, the stored data source is read and processed in parallel, thereby decomposing the stored data source into multiple parallel execution parts, making full use of the computing power of multi-core processors, significantly improving the processing speed and system throughput, and reducing the system's real-time processing time.

[0112] A corresponding data lake table is established based on the data source to be processed, and the data source and the data lake table are associated. This allows for complete association of data source information with the data lake table, facilitating filtering of the data source based on user business tag information in the data lake table. The data lake table provides high throughput, low latency data ingestion, and real-time query capabilities, improving the speed of real-time data processing. By determining the full set of rule data from the data source to be processed, the rule information of the data source can be extracted more easily, improving the filtering speed during real-time data processing. Based on the user business rule information corresponding to the rule broadcast stream, the results of the association processing between the data source and the data lake table are filtered. This achieves real-time rule matching, dynamic configuration updates, and global state synchronization through broadcast streams, providing fast filtering speed for real-time data processing and improving the speed and efficiency of real-time data processing.

[0113] Figure 3 This is a schematic diagram of the structure of a business data processing device provided in an embodiment of this application. Figure 3 As shown, the business data processing apparatus 300 provided in this application includes:

[0114] The acquisition module 301 is used to acquire the data source to be processed, which includes user information and business data, and to perform association processing between the business data and the data lake table based on the user information to obtain a first association result. The data lake table is used to indicate the table corresponding to the tag information of the business data.

[0115] Processing module 302 is used to filter the first association result according to the rule broadcast stream to obtain the target association result, wherein the rule broadcast stream is used to indicate the filtering rules corresponding to the business data;

[0116] The push module 303 is used to determine the corresponding target business data based on the target association result, and to push the target business data.

[0117] Optionally, the processing module 302 is further configured to store the data source into a memory cache based on the user information of the data source obtained in real time, and obtain the storage result;

[0118] The processing module 302 is further configured to perform parallel reading processing on the stored result based on the processing operator of the stream processing framework to obtain the data source to be processed after parallel reading processing. The processing operator includes multiple degrees of parallelism, and the parallel reading processing is performed according to the multiple degrees of parallelism. Each degree of parallelism contains corresponding unique identification information.

[0119] Optionally, the processing module 302 is further configured to establish a corresponding data lake table based on the data source to be processed, wherein the data lake table includes: user business tag information;

[0120] The processing module 302 is further configured to associate the data source to be processed with the data lake table according to the user information to obtain a first association result. The first association result includes the user business tag information and corresponds to the rule information of the rule broadcast stream for filtering.

[0121] Optionally, the device further includes: a determining module 304;

[0122] The determining module 304 is used to determine full rule data based on the data source to be processed, wherein each rule data in the full rule data corresponds to the rule information of the data source to be processed, and the rule information includes: user business rule information.

[0123] Optionally, the determining module 304 is further configured to determine the first rule broadcast stream corresponding to the first association result;

[0124] The processing module 302 is further configured to filter the first association result according to the first rule broadcast stream to obtain the target association result.

[0125] Optionally, the determining module 304 is further configured to determine the user service rule information corresponding to the first rule broadcast stream;

[0126] The processing module 302 is further configured to filter the user service tag information corresponding to the first association result based on the user service rule information corresponding to the first rule broadcast stream, so as to obtain the target association result.

[0127] Optionally, the push module 303 is further configured to store the target business data into the stream processing platform so that the stream processing platform can perform real-time push processing on the target business data.

[0128] Figure 4 This is a schematic diagram of the structure of a business data processing device provided in this application. Figure 4 As shown, this application provides a business data processing device 400, which includes a receiver 401, a transmitter 402, a processor 403, and a memory 404.

[0129] Receiver 401 is used to receive instructions and data;

[0130] Transmitter 402 is used to send commands and data;

[0131] Memory 404 is used to store instructions executed by the computer;

[0132] The processor 403 is used to execute computer execution instructions stored in the memory 404 to implement the various steps of the business data processing method in the above embodiments. For details, please refer to the relevant descriptions in the foregoing embodiments of the business data processing method.

[0133] Optionally, the memory 404 can be either standalone or integrated with the processor 403.

[0134] When the memory 404 is set up independently, the electronic device also includes a bus for connecting the memory 404 and the processor 403.

[0135] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the business data processing method performed by the aforementioned business data processing device.

[0136] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0137] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0138] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A business data processing method, executed based on a stream processing framework, characterized in that, The method includes: Obtain the data source to be processed, which is number portability service data, and the data source to be processed includes: user number information and service data; A corresponding data lake table is established based on the data source to be processed. The data lake table includes: user business tag information; the user business tag information includes one or more of the following: user group information, user age information, user gender information, user number portability time information, user porting in and out network information, user original network information, and user home location field information; the data lake table is used to indicate the table corresponding to the tag information of the business data. The data source to be processed is associated with the data lake table based on the user number information to obtain a first association result, which includes the user business tag information. Determine the first rule broadcast stream corresponding to the first association result; based on the first rule broadcast stream and using a dynamic expression evaluation engine, filter the user business tag information corresponding to the first association result to obtain the target association result; Based on the target association results, the corresponding target business data is determined, and the target business data is pushed to the system.

2. The method according to claim 1, characterized in that, Before obtaining the data source to be processed, the method further includes: Based on the user information obtained from the data source in real time, the data source is stored in the memory cache to obtain the storage result; The processing operator based on the stream processing framework performs parallel reading processing on the stored results to obtain the data source to be processed after parallel reading processing. The processing operator includes multiple degrees of parallelism, and the parallel reading processing is performed according to the multiple degrees of parallelism. Each degree of parallelism contains corresponding unique identification information.

3. The method according to claim 1, characterized in that, Before filtering the first association result according to the rule broadcast stream to obtain the target association result, the method further includes: The full set of rule data is determined based on the data source to be processed. Each rule in the full set of rule data corresponds to the rule information of the data source to be processed. The rule information includes user business rule information.

4. The method according to claim 3, characterized in that, The step of filtering the first association result according to the first rule broadcast stream to obtain the target association result includes: Determine the user service rule information corresponding to the first rule broadcast stream; Based on the user business rule information corresponding to the first rule broadcast stream, the user business tag information corresponding to the first association result is filtered to obtain the target association result.

5. The method according to claim 1, characterized in that, The process of pushing the target business data includes: The target business data is stored in the stream processing platform so that the stream processing platform can push and process the target business data in real time.

6. A business data processing apparatus, based on a stream processing framework, characterized in that, include: The acquisition module is used to acquire a data source to be processed, which is number portability service data. The data source to be processed includes: user number information and service data; to establish a corresponding data lake table based on the data source to be processed, the data lake table including: user service tag information; the user service tag information includes one or more of the following: user group information, user age information, user gender information, user number portability time information, user porting in and out network information, user original network information, and user home location field information; the data lake table is used to indicate the table corresponding to the tag information of the service data; and to perform association processing between the data source to be processed and the data lake table based on the user number information to obtain a first association result, the first association result including the user service tag information. The determination module is used to determine the first rule broadcast stream corresponding to the first association result; The processing module is used to broadcast the stream according to the first rule and use a dynamic expression evaluation engine to filter the user business tag information corresponding to the first association result to obtain the target association result. The push module is used to determine the corresponding target business data based on the target association result, and to push the target business data.

7. A business data processing device, characterized in that, include: Memory; processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the business data processing method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the business data processing method as described in any one of claims 1-5.

Images