Message data filtering method and device, storage medium and electronic device

Abstract

The invention discloses a message data filtering method and device, a storage medium, an electronic device and a computer program product. The method comprises the following steps: collecting message customer group data from a plurality of different message channels; carrying out distributed storage on the collected message customer group data by adopting a distributed file system; selecting corresponding filtering conditions from a preset strategy library according to the target business demand for combination so as to determine a filtering strategy; and filtering the distributed stored message customer group data by adopting a filtering strategy to obtain target customer group data, the target customer group data being used for being pushed to the target object. The filtering strategies are flexibly combined based on service requirements, and the data are quickly filtered by adopting a distributed processing technology, so that efficient, accurate and flexible processing of large-scale message data is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information filtering, and in particular, to a message data filtering method and device, a storage medium, an electronic device, and a computer program product. BACKGROUND

[0002] With the deepening of digital transformation and the rapid development of Internet technology, enterprises need to process a large amount of message data from different channels every day, such as social media, short message service, mobile application, etc. These data are not only large in quantity, but also diverse in type, covering multi-dimensional information such as user behavior, preference, feedback, etc. Efficient and accurate processing and analysis of these message data is crucial for improving customer experience, optimizing marketing strategies, and enhancing data analysis capabilities.

[0003] In related technologies, a fixed filtering method is usually used to filter data.

[0004] However, the related art method is usually inflexible and personalized, and cannot meet the specific needs of different customer groups or different businesses. SUMMARY

[0005] The present application provides a message data filtering method and device, a storage medium, an electronic device, and a computer program product.

[0006] According to an aspect of an embodiment of the present application, a message data filtering method is provided, which includes: collecting message customer group data from a plurality of different message channels; using a distributed file system to distribute the collected message customer group data; determining the type of required filtering conditions according to the target business requirements; selecting the corresponding type of filtering conditions from the pre-set condition library according to the type of required filtering conditions, to obtain a first condition set; determining the priority of the plurality of filtering conditions in the first condition set according to the historical execution effect of each filtering condition in the first condition set; determining the filtering order of the plurality of filtering conditions according to the priority of the plurality of filtering conditions; combining the plurality of filtering conditions according to the filtering order to obtain the filtering strategy; and filtering the distributed message customer group data using the filtering strategy to obtain target customer group data, wherein the target customer group data is used to push to target objects.

[0007] In an example embodiment, the determining the filtering sequence of the plurality of filtering conditions according to the priority of the plurality of filtering conditions comprises: determining an initial filtering sequence by sorting the plurality of filtering conditions according to the priority of the plurality of filtering conditions; determining data density of different types of message crowd data according to data types and data volume of the message crowd data, wherein the data types comprise text, image, and video; respectively simulating expected redundant data generated when each of the filtering conditions is used to filter the message crowd data according to the data density of each type of message crowd data and filtering rules of the plurality of filtering conditions; determining the consumption of computer resources when each of the filtering conditions is executed according to the expected redundant data corresponding to each of the filtering conditions; adjusting the initial filtering sequence according to the consumption of computer resources when each of the filtering conditions is executed and an execution dependency relationship between the plurality of filtering conditions to obtain the filtering sequence of the plurality of filtering conditions.

[0008] In an example embodiment, the method further comprises: obtaining identity information and historical behavior data of the target object; determining a behavior pattern and a preference condition of the target object according to the identity information and the historical behavior data of the target object, wherein the preference condition comprises an active time period of the target object and an on-off setting for information recommendation; adjusting the filtering condition according to the behavior pattern and the preference condition of the target object to update the filtering strategy.

[0009] In an example embodiment, the message crowd data collected is stored in a distributed manner using a distributed file system, comprising: deploying a distributed storage cluster according to expected data volume and data processing requirements of the message crowd data, wherein the storage cluster comprises a plurality of storage nodes; deploying a management component on the plurality of nodes in the storage cluster, wherein the management component is used to manage metadata and computing resources in the distributed file system; configuring parameter settings for storing data in the distributed file system based on the management component, wherein the parameters comprise data block size, number of copies, and storage directory structure; dividing the collected message crowd data into a plurality of data blocks according to the parameter settings, and replicating the plurality of data blocks to obtain a plurality of copies; dispersively storing the plurality of data blocks and the plurality of copies on the plurality of storage nodes.

[0010] In an example embodiment, after dispersively storing the plurality of data blocks and the plurality of copies on the plurality of storage nodes, the method further comprises: establishing a database table using a warehouse data tool corresponding to the distributed file system, wherein the database table is pre-defined with fields, data types, and partitioning strategies; importing the crowd data stored in the distributed file system into the database table to establish a mapping relationship in the database table, wherein the mapping relationship is used to indicate the correspondence between the storage location of the crowd data and the directory of the crowd data in the database table.

[0011] In an example embodiment, after the collected message customer data is stored in a distributed manner by using the distributed file system, the method further comprises: establishing an external table by using a warehouse data tool of the distributed file system, wherein the external table is associated with a storage location of the message customer data in the distributed file system; loading the message customer data in the distributed file system into the external table; identifying abnormal data in the message customer data in the external table and cleaning the abnormal data; converting the cleaned message customer data into a preset data format; and performing standardization processing on the message customer data in the preset data format.

[0012] According to another aspect of the embodiments of the present application, a message data filtering device is further provided, which comprises:

[0013] a data collection module, configured to collect message customer data from a plurality of different message channels;

[0014] a data storage module, configured to store the collected message customer data in a distributed manner by using a distributed file system;

[0015] a strategy determination module, configured to determine a type of required filtering condition according to a target business requirement; select a filtering condition of a corresponding type from a preset condition library according to the type of the required filtering condition, to obtain a first condition set; determine a priority of a plurality of filtering conditions in the first condition set according to a historical execution effect of each filtering condition in the first condition set; determine a filtering order of the plurality of filtering conditions according to the priorities of the plurality of filtering conditions; and combine the plurality of filtering conditions according to the filtering order, to obtain the filtering strategy;

[0016] a filtering module, configured to filter the message customer data stored in a distributed manner by using the filtering strategy, to obtain target customer data, wherein the target customer data is used for pushing to a target object.

[0017] According to still another aspect of the embodiments of the present application, a computer readable storage medium is further provided, wherein the computer readable storage medium stores a computer program, and the computer program is configured to execute the message data filtering method when running.

[0018] According to still another aspect of the embodiments of the present application, an electronic device is further provided, which comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the message data filtering method by using the computer program.

[0019] According to a further aspect of the embodiments of the present application, a computer program product is also provided, which comprises a computer program, and the computer program, when executed by a processor, implements the steps of the method described in various embodiments of the present application.

[0020] The filtering method of message data described above collects message data from multiple channels, stores the data efficiently using a distributed file system, flexibly combines filtering strategies based on business needs, and quickly filters data using distributed processing technology. The business needs and the type of filtering conditions are accurately matched, ensuring the relevance and effectiveness of the filtering strategies. The existence and flexible selection of the condition library enable the filtering strategies to quickly adapt to market changes and meet diverse business needs. By analyzing historical effects to determine the priority and order of filtering conditions, the efficiency and quality of data filtering are improved. The logical combination and order adjustment mechanism ensures the customization and flexibility of the filtering strategies, allowing real-time adjustments based on business needs. By analyzing target business needs, flexibly selecting and combining filtering conditions, and optimizing filtering order, efficient and accurate filtering of message customer data is achieved. This effectively improves the efficiency and effectiveness of data processing, reduces resource consumption, enhances the flexibility and adaptability of the system, and provides strong support for data-driven decision-making. Efficient, accurate, and flexible processing of large-scale message data is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate an embodiment consistent with the present application and, together with the description, serve to explain the principles of the application.

[0022] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative labor.

[0023] Figure 1 is a hardware structure block diagram of the filtering method of message data according to an embodiment of the present application;

[0024] Figure 2 is a flowchart of a filtering method of message data according to an embodiment of the present application;

[0025] Figure 3 is a flowchart of a filtering method of message data according to an embodiment of the present application;

[0026] Figure 4 is a flowchart of a filtering method of message data according to an embodiment of the present application;

[0027] Figure 5Fig. 4 is a flowchart of a fourth message data filtering method according to an embodiment of the present application;

[0028] Figure 6 Fig. 5 is a flowchart of a fifth message data filtering method according to an embodiment of the present application;

[0029] Figure 7 Fig. 6 is a flowchart of a sixth message data filtering method according to an embodiment of the present application;

[0030] Figure 8 Fig. 7 is a schematic block diagram of a message data filtering method according to an embodiment of the present application;

[0031] Figure 9 Fig. 8 is a structural block diagram of a message data filtering device according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative work should fall within the scope of protection of the present application.

[0033] It should be noted that the terms “first”, “second”, and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms “include” and “have” and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0034] The method embodiments provided in the embodiments of the present application can be executed in a computer terminal or similar computing device. Taking a computer terminal as an example, Figure 1 Fig. 9 is a hardware structural block diagram of a computer terminal of a message data filtering method according to an embodiment of the present application. As shown in Figure 1 the computer terminal can include one or more (such as one) central processing units (CPU) 901, memories 902, buses 903, input / output (I / O) interfaces 904, and storage devices 905. Figure 1The computer terminal shown in the figure only includes one processor 102 (the processor 102 can include, but is not limited to, a microprocessor unit (MPU) or a programmable logic device (PLD)) and a memory 104 for storing data. In an exemplary embodiment, the computer terminal can further include a transmission device 106 for communication function and an input / output device 108. Those skilled in the art can understand that, Figure 1 The structure shown in the figure is only schematic and does not limit the structure of the computer terminal. For example, the computer terminal can include more or fewer components than those shown in the figure, or have a different configuration with the same or more functions than those shown in the figure. Figure 1 Figure 1 Figure 1

[0035] The memory 104 can be used to store computer programs, such as software programs of application software and modules, such as a computer program corresponding to the message data filtering method in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, that is, implements the above-mentioned method. The memory 104 can include a high-speed random access memory and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely arranged with respect to the processor 102, and these remote memories can be connected to the computer terminal through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0036] The transmission device 106 is used to receive or send data via a network. Specific examples of the network can include a wireless network provided by a communication provider of the computer terminal. In an example, the transmission device 106 includes a network adapter (NIC) which can be connected to other network devices through a base station so as to communicate with the Internet. In an example, the transmission device 106 can be a radio frequency (RF) module which is used to communicate with the Internet in a wireless manner.

[0037] In the technical solution of the present application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data comply with relevant laws and regulations and do not violate public order and good customs.

[0038] ​​​It should be noted that in the embodiments of the present application, some industry existing solutions such as certain software, components, models, etc. may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility in the implementation of the technical solutions of the present application, but does not mean that the applicant has or will necessarily use the solutions.

[0039] In the present embodiment, a message data filtering method is provided, Figure 2 is a flowchart of an optional message data filtering method according to an embodiment of the present application, which comprises the following steps S200-S270:

[0040] Step S200, collecting message customer group data from multiple different message channels.

[0041] Specifically, in today's highly interconnected business environment, it is necessary to process message data from multiple channels, including but not limited to social platforms, short message services, email systems, mobile applications, etc. This step is the starting point of the big data message processing flow, and through the integration of the application programming interface (API) interfaces of each message source, the user messages and behavior data scattered in different platforms are collected and centralized, laying a foundation for subsequent data processing and analysis.

[0042] Exemplarily, a message collection system is developed, which can establish a connection with multiple message channels, listen to and capture message data in real time, and the channels include business centers, marketing centers, customer group databases, etc.

[0043] Step S210, using a distributed file system to distribute the collected message customer group data for distributed storage.

[0044] Specifically, the collected message customer group data is often large and continuously growing, and using traditional centralized storage systems not only cannot meet the storage requirements, but also will face problems of data access delay and storage scalability. Therefore, this step uses a distributed file system (such as Hadoop Distributed File System (HDFS)) to store the data on multiple nodes, ensuring high reliability and high-speed access of the data. By collecting message data from multiple different channels, a comprehensive user behavior view can be obtained, providing data support for fine operation. At the same time, centralized data collection can also help to simplify data management, reduce data island phenomenon, and improve data availability and consistency.

[0045] Exemplarily, HDFS is deployed on a Hadoop cluster, and the collected message customer data is uploaded to HDFS. HDFS automatically splits the data into fixed-size data blocks (default 128 MB) and replicates (usually 3 copies) these data blocks on different nodes of the cluster to improve data reliability. At the same time, using the directory structure function of HDFS, a corresponding subdirectory is created under the / user / hive / warehouse directory to store the processed data, facilitating subsequent querying and analysis. Distributed storage ensures high throughput data read-write capability and high data reliability. The horizontal expansion feature of HDFS means that as the amount of data grows, storage capacity can be easily expanded by adding Hadoop nodes without causing system performance degradation. The multiple-copy storage of data effectively prevents data loss, and even if some nodes fail, continuous access to data is guaranteed.

[0046] Step S220, determine the type of filtering condition required according to the target business requirement.

[0047] Specifically, in the context of big data analysis and processing, it is necessary to determine which filtering conditions are most critical to filter out the most relevant target customer groups according to specific business goals. This process ensures the relevance and efficiency of data processing, avoiding unnecessary calculation and storage resource waste. By accurately matching business requirements and filtering condition types, the accuracy and effectiveness of subsequent data processing can be ensured, improving the efficiency of the overall business process and reducing the occupation of resources by irrelevant data.

[0048] Exemplarily, a business requirement analysis module is set up, which determines the key condition types for filtering message customer data based on current marketing strategies, compliance requirements, customer preferences, and other information through manual input or machine learning prediction. For example, if the business requirement is to improve the opening rate of marketing emails, filtering conditions such as "user activity" and "user preferences" may need to be considered.

[0049] Step S230, select the filtering conditions of the corresponding type from the pre-set condition library according to the type of the required filtering conditions, and obtain a first condition set.

[0050] Specifically, according to the filtering condition type determined in the previous step, specific filtering conditions are selected from the pre-constructed condition library to form a condition set for subsequent data filtering. This step aims to ensure the flexibility and diversity of filtering rules to adapt to the needs of different business scenarios.

[0051] Exemplarily, a condition library is constructed to store various types of filtering conditions, such as user authorization status, user activity, user preferences, and the like. When the system determines the type of filtering condition, it selects a filtering condition matching the type from the condition library to form a first condition set, preparing conditions for subsequent filtering strategies. The establishment of the condition library enables a quick response to market changes and business needs, and through simple selection and combination of filtering conditions, a customized filtering strategy can be constructed, greatly improving the flexibility and customization capability of data processing.

[0052] In step S240, the priority of each filtering condition in the first condition set is determined according to the historical execution effect of each filtering condition in the first condition set.

[0053] Specifically, considering that filtering conditions may have different effects on data processing efficiency and result quality, this step aims to determine the priority of each filtering condition by analyzing historical data to optimize the data filtering process and reduce the waste of computing resources. By determining the priority of filtering conditions through historical execution effects, the efficiency and effectiveness of data filtering can be significantly improved, reducing unnecessary data processing steps, thereby reducing computing costs and improving data processing accuracy.

[0054] Exemplarily, an effect evaluation module is developed to analyze the effect of each filtering condition in past executions, including key indicators such as data reduction ratio, processing time, user feedback, and the like. According to these indicators, a priority is assigned to each filtering condition, and conditions with high priority will be executed first.

[0055] In step S250, the filtering order of the multiple filtering conditions is determined according to the priority of the multiple filtering conditions.

[0056] Specifically, after determining the priority of the filtering conditions, the next step is to determine the execution order of the filtering according to the priority order. This step is crucial for optimizing the data processing flow and reducing resource consumption. A reasonable filtering order can reduce the total time of data processing, improve the efficiency of data filtering, and at the same time ensure that the filtering results meet high-quality standards, providing reliable data support for subsequent business decisions.

[0057] Exemplarily, a filtering order manager is implemented to determine which conditions should be applied first and which conditions can be applied later based on the priority order of the filtering conditions. The determination of the filtering order is based on the degree of reduction of the data volume and the impact on the quality of the filtering results. The filtering order is the order of execution of different conditions in the data filtering process, and the optimization of the order can significantly improve the efficiency and quality of data processing.

[0058] In step S260, the multiple filtering conditions are combined according to the filtering order to obtain a filtering strategy.

[0059] Specifically, the filtering conditions are combined in a determined order to form a complete filtering strategy. The combination of strategies can be flexibly adjusted to adapt to different business needs and data environments. The dynamically generated filtering strategy ensures the flexibility and customization of data filtering, which can meet the changing business needs, while improving the accuracy and precision of data processing through logical combination.

[0060] Illustratively, a strategy combination engine is designed, which combines the filtering conditions in the first condition set according to the filtering order through logical operators (such as AND, OR) to generate a filtering strategy. The strategy combination engine should also provide an interface to allow users to adjust the strategy details as needed, such as adding new filtering conditions or changing the logical relationship between conditions.

[0061] Specifically, in big data processing, the filtering strategy is the key to screening and simplifying data, which can help extract valuable information from massive data. According to business goals and user preferences, filtering conditions are selected and combined from the strategy library to generate a filtering strategy suitable for current business needs.

[0062] Illustratively, a strategy management platform is designed, which integrates common filtering conditions such as user authorization status, subscription status, periodic sending quota, and protection of minors. When new business needs arise, such as launching a preferential activity only for users who have signed up and agreed to receive marketing information, the strategy management platform can select the "user authorization" and "subscription status" conditions and combine them into a new filtering strategy using logical operators. The strategy management platform also supports real-time updating and adjusting of filtering conditions to ensure that the strategy can quickly adapt to changes in the business environment. Through the establishment of a strategy library and the flexible combination of strategies, the platform can quickly respond to market changes and ensure the accuracy and compliance of message pushing. The dynamic adjustment mechanism of the strategy also reduces the data processing delay caused by changes in business policies, improving the efficiency and flexibility of data processing.

[0063] Illustratively, filtering conditions can include: user authorization filtering: ensures that only messages with user authorization can be processed and sent, avoiding illegal message pushing. Subscription status filtering: checks the subscription status of the message sending object, excluding users who have not signed up or have canceled the contract. Periodic quota filtering: limits the number of messages received by each user according to the set time period (such as day, week, month), preventing excessive disturbance. Minors filtering: filters out messages unsuitable for minors based on user-provided age information. Personalized recommendation filtering: analyzes user historical behavior and interests to ensure that the pushed messages match the user's preferences. Do-not-disturb period filtering: automatically filters message pushing during the do-not-disturb period set by the user. Blacklist filtering: prevents users on the blacklist from receiving any messages, protecting the security and user experience of the system.

[0064] At step S270, the message customer group data stored in the distributed manner is filtered by using the filtering strategy to obtain target customer group data.

[0065] The target customer group data is used for pushing to target objects.

[0066] Specifically, after the filtering strategy is determined, the next step is to filter the message customer group data stored in the HDFS, and screen out the user group meeting the filtering condition as the target customer group for subsequent message pushing.

[0067] Exemplarily, the filtering strategy can be implemented by using the SQL-like query language (HiveQL) of Hive. For example, for the combined condition of "user authorization" and "subscription status", the following query can be written: "SELECT FROM messagesWHERE authorized=TRUE AND subscribed=TRUE". The query is converted into a MapReduce task by the Driver component of Hive, and is run in parallel on the Hadoop cluster to filter the data in the HDFS. The customer group data obtained after the filtering is stored in the form of a Hive table, facilitating subsequent analysis and message pushing. With the distributed processing capability of Hive and Hadoop, the filtering of the message customer group data becomes efficient and scalable. The parallel computing feature of MapReduce greatly shortens the data processing time, and can process PB-level data volume. Meanwhile, the SQL-like query language provided by Hive simplifies the complexity of strategy implementation, and makes the definition and execution of the filtering strategy more intuitive and convenient.

[0068] In this embodiment, by collecting message data from multiple channels, using a distributed file system for efficient storage, flexibly combining filtering strategies based on business needs, and using distributed processing technology for rapid data filtering, the business needs and filtering condition types are accurately matched, ensuring the relevance and effectiveness of the filtering strategy. The existence and flexible selection of the condition library enable the filtering strategy to quickly adapt to market changes and meet diverse business needs. By analyzing historical results to determine the priority and order of filtering conditions, the efficiency and quality of data filtering are improved, and invalid calculations are reduced. The logical combination and order adjustment mechanism ensures the customization and flexibility of the filtering strategy, allowing real-time adjustments based on business needs. By analyzing target business needs, flexibly selecting and combining filtering conditions, and optimizing filtering order, efficient and accurate filtering of message customer data is achieved. This effectively improves data processing efficiency and effectiveness, reduces resource consumption, enhances system flexibility and adaptability, and provides strong support for data-driven decision-making. Efficient, accurate, and flexible processing of large-scale message data is achieved.

[0069] In one embodiment, as shown in Figure 3 Step S250, the filtering order of the plurality of filtering conditions is determined according to the priority of the plurality of filtering conditions. This includes steps S300-S340:

[0070] Step S300, the plurality of filtering conditions are sorted according to the priority order of the plurality of filtering conditions to determine the initial filtering order.

[0071] Specifically, the filtering conditions are sorted according to their priority to preliminarily determine the execution order of data filtering. Such sorting can ensure that in the early stages of data processing, those filtering conditions that can reduce invalid data or have the greatest impact on the final results are executed first.

[0072] Illustratively, the strategy management platform obtains historical execution data of the filtering conditions from the database, including the efficiency and effectiveness of each filtering condition in past data processing. Through calculation and analysis, a priority value is assigned to each filtering condition, with a higher priority value indicating that the condition is more important and should be executed first. All filtering conditions are sorted by priority value from high to low to form an initial filtering order list. Determining the priority of the filtering conditions and sorting them can significantly reduce invalid operations during data processing, improving the efficiency and resource utilization of data processing. Because filtering conditions with high priority can quickly filter out most of the data that does not meet the conditions, reducing the amount of data for subsequent processing, and thus speeding up the entire data processing process.

[0073] wherein the filtering condition priority is determined according to the performance of the filtering condition in historical data processing, including efficiency in reducing data volume, contribution to business goals, and other factors.

[0074] Step S310, according to the data type and data volume of the message crowd data, determine the data density of different types of message crowd data.

[0075] Among them, the data type includes text, image, video.

[0076] Specifically, analyze the type and volume of message crowd data, and determine the density of each type of data. The analysis of data density helps to understand the composition characteristics of data, which is crucial for subsequent filtering rule selection and resource allocation.

[0077] Illustratively, the data processing module reads the message crowd data collected from different channels, identifies the data type, and distinguishes between text, image, video and other forms of information. For each type of data, calculate its average size and total volume, and then obtain the data density. Data density can be determined by comprehensive analysis of metadata related to data volume and type (e.g. image resolution, video length). By analyzing the type and volume of data, its data density is determined, which helps to more reasonably configure the execution strategy of filtering conditions. Types with higher data density may require more computing resources or more refined filtering strategies, and vice versa. This can ensure efficient allocation of resources and avoid excessive resource investment for simple data types, thereby improving the processing efficiency and economy of the entire system.

[0078] Among them, the data density refers to the amount of data or the density of information in a unit of storage space, which can reflect the complexity and storage efficiency of data.

[0079] Step S320, according to the data density of each type of message crowd data, and the filtering rules of multiple filtering conditions, respectively simulate the expected redundant data generated when each filtering condition is used to filter the message crowd data.

[0080] Specifically, based on data density and filtering rules, predict the unnecessary data (redundant data) that may be generated when each filtering condition is executed. This step helps to further optimize the execution order of filtering conditions and reduce resource waste.

[0081] Illustratively, use statistical models or empirical formulas to predict the remaining proportion of data after filtering according to the data type and data volume. Simulate the execution process of each filtering condition, combine the filtering rules and data density, and calculate the expected amount of data after filtering. Subtract the difference between the amount of data before filtering and the amount of data after filtering from the amount of data after filtering to estimate the amount of redundant data. By predicting the redundant data that may be generated by each filtering condition, the impact of filtering operation on system resources can be more accurately evaluated. Reducing redundant data means reducing unnecessary storage and computing needs, which helps to reduce operating costs and improve data processing efficiency and speed.

[0082] wherein the redundant data is data that is determined to be no longer needed or of no practical value for subsequent processing during data processing.

[0083] At step S330, the consumption of computer resources by each filter condition when executed is determined based on the predicted redundant data corresponding to each filter condition.

[0084] Specifically, the actual demand for computer resources when executing the filter conditions is analyzed, including CPU time, memory usage, and disk I / O, etc., in order to more accurately manage resources and optimize resource allocation.

[0085] Illustratively, a resource consumption model is established for each filtering rule, and the resource consumption is estimated by analyzing its algorithmic complexity and data processing requirements. In combination with the predicted amount of redundant data, the actual resource consumption when executing each filter condition is calculated.

[0086] wherein the computer resource consumption is the computing resources that need to be allocated by the system when executing the filtering operation, such as CPU clock cycles, memory usage, disk input / output operation times, etc.

[0087] At step S340, the initial filtering order is adjusted based on the consumption of computer resources by each filter condition when executed, as well as the execution dependency relationship between multiple filter conditions, to obtain a filtering order of the multiple filter conditions.

[0088] Specifically, the initial filtering order is adjusted in combination with the resource consumption of the filter conditions and their dependency relationship, to obtain a final optimized filtering order. This order can maximize the efficiency of data processing while ensuring the correct execution of the filtering rules.

[0089] Illustratively, based on the resource consumption, filter conditions with small resource consumption are preferentially arranged to be executed first. Considering the dependency relationship of the filter conditions, it is ensured that the results of upstream conditions are the premise of the execution of downstream conditions. By comprehensively considering the priority, resource consumption, and dependency relationship, the execution order of the filter conditions is rearranged using graph theory or optimization algorithms to form the filtering strategy of the present application. By adjusting the filtering order, the system can more effectively allocate and utilize computing resources, while also ensuring the correct execution of the filtering rules. This not only improves the overall efficiency of data processing, but also avoids performance bottlenecks caused by excessive concentration of resources on certain filter conditions, thereby achieving dual optimization of resource consumption minimization and data processing effect maximization.

[0090] wherein the execution dependency relationship refers to the front-back correlation between the filter conditions, and the execution results of some conditions may affect the effectiveness or execution efficiency of other conditions.

[0091] In this embodiment, from determining the priority of filtering conditions, analyzing data density, to simulating redundant data, evaluating resource consumption, and finally adjusting the filtering order, a complete optimization process is formed. The core of this process is to efficiently utilize resources and reduce redundant operations, ensuring the accuracy and rapid response capability of data processing. Specifically, through priority determination and initial sorting, the system can quickly eliminate a large amount of irrelevant data; through data density analysis, the system can more reasonably allocate computing resources; simulation of redundant data and resource consumption evaluation further refines the resource management strategy, avoiding unnecessary resource waste; and in the final adjustment of the filtering order, the system considers multiple factors to achieve the optimization path of the execution of filtering conditions, thereby ensuring data processing quality while improving processing speed and resource utilization. This scheme not only can handle large-scale message data, but also can dynamically adjust according to business needs and environmental changes, improving the flexibility and adaptability of the system.

[0092] In one embodiment, as shown in FIG. 4, the method further includes steps S400-S420: Figure 4

[0093] Step S400, obtaining the identity information and historical behavior data of the target object.

[0094] Specifically, in the field of big data analysis, understanding the detailed information of the target object is crucial for accurate push and personalized service. This step focuses on collecting the identity information (such as age, gender, geographic location, etc.) and historical behavior data (such as purchase records, browsing preferences, etc.) of the target object from various sources to build a comprehensive portrait of the user, laying the foundation for subsequent personalized analysis.

[0095] Illustratively, develop a user information collection system that can collect information from internal databases (such as Customer Relationship Management (CRM) systems, transaction databases) and external data sources (such as social media platforms, public data sets). Through API calls, data mining techniques or crawler programs, integrate the target object's data into a unified data lake to ensure real-time and complete data.

[0096] Step S410, determining the behavior pattern and preference situation of the target object according to the identity information and historical behavior data of the target object.

[0097] Among them, the preference situation includes the active time period of the target object and the on-off setting for information recommendation.

[0098] ​Specifically, after collecting sufficient user data, the next step is to determine the user's behavior patterns and preferences by analyzing these data. This step is crucial for achieving personalized services and intelligent recommendations, which can better meet user needs and improve user experience. Through in-depth analysis of user data, user needs and preferences can be better understood, personalized content and services can be provided to users, and user satisfaction and loyalty can be enhanced. In addition, the identification of behavior patterns can help predict future user behavior and optimize business processes.

[0099] For example, machine learning algorithms or statistical analysis methods such as clustering analysis, sequence models, decision trees, etc. are used to analyze the identity information and historical behavior data of the target object in depth. For example, by analyzing the user's browsing records and purchase history, the user's active time period and product interest preferences are determined. The analysis results are summarized to form the user's behavior patterns and preference profiles.

[0100] Among them, the behavior pattern is the regularity of the user's repeated behavior in a specific environment, such as daily routine, shopping habits, information consumption patterns, etc. The preference situation is the user's inclination or preference for a particular thing, such as favorite product type, preferred information type, time preference for receiving information, etc.

[0101] Step S420, according to the behavior pattern and preference situation of the target object, adjust the filtering condition, update the filtering strategy.

[0102] Specifically, after determining the user's behavior patterns and preferences, the enterprise needs to adjust the data filtering strategy to ensure that the information pushed not only meets the user's interests, but also respects the user's privacy and time preferences. This step dynamically adjusts the filtering rules to achieve the dual goals of personalization and compliance. Dynamic adjustment of filtering conditions can significantly improve the level of personalization of information pushing, reduce disturbance to users, and at the same time ensure the quality and relevance of the pushed content. This not only helps to enhance user satisfaction, but also improves the effective communication rate of marketing or service information, thereby improving the business effectiveness of the enterprise.

[0103] For example, according to the behavior pattern and preference situation of the target object, the conditions and parameters in the filtering strategy are automatically or semi-automatically adjusted. For example, if the analysis results show that the user is most active during 9pm to 11pm, then the exclusion time period in the "do not disturb time period" filtering condition can be set to other times except 9pm to 11pm. In addition, according to the user's on-off setting for information recommendation, the personalized recommendation filtering condition can be turned on or off.

[0104] In this embodiment, first, comprehensive user information is obtained, ensuring the comprehensiveness and depth of subsequent analysis, which is a prerequisite for personalized services. Second, the user data is deeply analyzed to identify the user's behavior patterns and preferences, providing the necessary basis for precise filtering. Finally, the filtering strategy is dynamically adjusted to achieve a balance between personalization and compliance, improving the effectiveness of information pushing and user satisfaction, while also considering the user's time preference and privacy protection. Through continuous collection, analysis, and adjustment, the accuracy and effectiveness of message pushing can be continuously improved, while reducing unnecessary resource consumption and interference to users.

[0105] In one embodiment, as shown in Figure 5 step S210, the collected message customer group data is distributedly stored using a distributed file system. This includes steps S500-S540:

[0106] Step S500, according to the expected data volume and data processing requirements of the message customer group data, a distributed storage cluster is deployed.

[0107] The storage cluster includes multiple storage nodes.

[0108] Specifically, according to the size of the message customer group data to be processed and the specific processing requirements, the data storage resources are reasonably planned and deployed. The distributed storage cluster stores data on multiple physical nodes to improve the efficiency, scalability, and fault tolerance of data processing. Reasonable planning and deployment of the storage cluster can ensure efficient access to data and meet the performance requirements of large-scale data processing, while distributed storage improves data reliability and overall system stability.

[0109] Illustratively, first, the data processing requirements are evaluated, including data read / write speed, data storage capacity, data processing delay requirements, etc. Then, according to the evaluation results, a suitable distributed file system such as Hadoop's HDFS is selected, and a sufficient number of storage nodes are deployed to meet the expected data processing requirements. For example, if the expected data volume is tens of PB level and real-time processing capability is required, it may be necessary to deploy a storage cluster consisting of hundreds or even thousands of servers.

[0110] Step S510, deploy management components on multiple nodes in the storage cluster.

[0111] The management component is used to manage the metadata and computing resources in the distributed file system.

[0112] Specifically, the management component is an important part of the distributed file system responsible for maintaining metadata and managing computing resources. By deploying the management component on each storage node, unified management and monitoring of the entire storage cluster can be achieved, ensuring data consistency and efficient processing. Through the deployment of the management component, not only can the correct storage and fast access of data be ensured, but also intelligent scheduling of resources can be achieved, avoiding resource waste and improving the efficiency of data processing and the overall performance of the system.

[0113] For example, in Hadoop's HDFS, NameNode and DataNode are deployed. NameNode serves as the master node, responsible for storing the metadata of the file system, such as file names, file attributes, block lists of files, and location information of blocks, etc. DataNode serves as the worker node, storing actual data block replicas. In addition, Resource Manager and NodeManager can be deployed to manage and schedule the computing resources of the entire cluster.

[0114] Step S520, configure the parameter settings of the distributed file system for storing data based on the management component.

[0115] The parameters include data block size, number of replicas, and storage directory structure.

[0116] Specifically, in order to ensure that data can be efficiently and reliably stored and accessed in a distributed storage cluster, the parameters for storing data need to be reasonably configured, such as data block size, number of replicas, and storage directory structure.

[0117] Step S530, divide the collected message customer group data into multiple data blocks according to the parameter settings, and replicate the multiple data blocks to obtain multiple replicas.

[0118] Specifically, to adapt to the storage characteristics of the distributed file system, this step will split the collected message customer group data into multiple data blocks and replicate multiple copies for each data block, and store them on different nodes in the cluster. The data block splitting and replica storage mechanism can significantly improve the data read speed and storage efficiency, while enhancing the data disaster recovery capability, ensuring that even if some nodes fail, data can still be quickly recovered and accessed.

[0119] Step S540, store the multiple data blocks and multiple replicas on multiple storage nodes.

[0120] Specifically, the segmented and replicated message group data blocks are then distributed across the nodes of a distributed storage cluster to achieve high availability and load balancing. Distributed storage of data block replicas not only improves read and write speeds but also achieves load balancing, reducing the risk of single points of failure and enhancing system scalability and fault tolerance. The distributed architecture allows the system to scale horizontally as data volume increases, adding processing power without replacing existing hardware.

[0121] In this embodiment, by rationally deploying a distributed storage cluster and configuring storage parameters, efficient storage and rapid access to message customer data are ensured, meeting the high-concurrency read / write requirements under large data volumes. Data block replication and distributed storage significantly enhance data redundancy and disaster recovery capabilities; even in the event of node failure, the system can quickly recover data, ensuring business continuity. The deployment of management components enables unified management of the storage cluster and computing resources, avoiding resource waste through intelligent scheduling and improving the overall performance and efficiency of the system. The distributed architecture inherently possesses horizontal scalability, easily adding storage nodes as data volume grows without downtime for upgrades, ensuring system flexibility and future expansion potential. By deploying and optimizing the distributed storage cluster, efficient, secure, and flexible storage of message customer data, as well as high-performance data processing, are achieved, providing a solid foundation for subsequent data analysis and processing, while also ensuring system stability and scalability. It is particularly suitable for scenarios handling large-scale message data, significantly improving data processing efficiency and reliability, reducing operating and maintenance costs, and meeting the needs for real-time and historical data analysis.

[0122] In one embodiment, such as Figure 6 As shown, after distributing multiple data blocks and multiple replicas across multiple storage nodes in step S540, the method further includes: steps S600-S610:

[0123] Step S600: Use the warehouse data tool corresponding to the distributed file system to create database tables.

[0124] The database tables are predefined with fields, data types, and partitioning strategies.

[0125] Specifically, a database table is created using a data warehouse tool compatible with the distributed file system (such as Apache Hive) to store and manage the customer data collected from multiple message channels. By defining the table structure, including fields, data types, and partitioning strategies, the storage and query efficiency of the data can be optimized. Through reasonable table structure and partitioning strategy design, the storage efficiency and query performance of the data can be significantly improved, especially when dealing with large-scale data sets. Partitioning can reduce the query range to a specific partition, greatly reducing the amount of data scanning and improving the query speed.

[0126] In step S610, the customer data stored in the distributed file system is imported into the database table, and a mapping relationship is established in the database table.

[0127] The mapping relationship is used to indicate the correspondence between the storage location of the customer data and the directory of the customer data in the database table.

[0128] Specifically, the data stored in the distributed file system (such as HDFS) needs to be imported into the database table defined earlier to take advantage of the powerful processing capabilities of the data warehouse tool for data analysis and filtering. At the same time, a mapping relationship is established between the storage location of the customer data in the distributed file system and the directory in the database table to ensure accurate positioning and efficient access of the data. Mapping relationship: in the data warehouse, it refers to the correspondence from the storage location of the distributed file system to the directory of the database table. This mapping allows the data warehouse to directly access the data in the storage system without the need to copy the data to another storage area. After establishing the mapping relationship, the data warehouse can directly perform data operations on the distributed file system, avoiding the need for secondary handling of data, reducing input / output (I / O) operations, saving storage space and processing time.

[0129] In this embodiment, by defining appropriate fields and data types, and implementing an efficient partitioning strategy, the efficiency of data storage and query performance are improved, especially when dealing with PB-level large data, the amount of data scanning can be significantly reduced, and the query speed can be accelerated. The data in HDFS is directly imported into the Hive table, and the mapping between the storage location and the database table is established, which simplifies the data access path, reduces the data handling, and reduces the overall delay of data processing. At the same time, the establishment of the mapping relationship enables the data warehouse to directly operate the data in the original storage location, improving the flexibility and efficiency of data processing. Using the data warehouse function of Hive, the data stored in HDFS can be efficiently processed without increasing the additional hardware burden. The SQL-like query language of Hive and the built-in data processing capability enable data analysts and developers to perform complex data operations without deep understanding of the underlying HDFS working principle, thereby reducing the skill threshold and improving the work efficiency. The distributed storage characteristics of HDFS and the high availability of Hive ensure the scalability and fault tolerance of data processing. Even if some nodes fail, the data can still be recovered from the copies of other nodes, ensuring the continuous availability of data. At the same time, by increasing the number of nodes, the rapid growth of data volume can be easily handled without frequent upgrading of hardware devices, thereby reducing the maintenance cost. By storing data on HDFS and managing data in Hive, data security and consistency can be guaranteed, and the multi-copy mechanism of HDFS ensures the persistent storage of data.

[0130] In one embodiment, as shown in FIG. 7, after the collected message customer data is distributedly stored by using the distributed file system in step S210, the method further comprises steps S700-S740: Figure 7

[0131] In step S700, a warehouse data tool of the distributed file system is used to establish an external table.

[0132] Specifically, an external table is created in a data warehouse tool (such as Apache Hive) to enable direct reference and operation of the message customer data stored in the distributed file system (such as Hadoop HDFS). The external table is associated with the actual data file in HDFS without moving the data to the internal storage of Hive, which is beneficial to the maintenance and update of data while preserving the original location of the data. By establishing an external table, the data query and management functions of Hive can be used without occupying additional storage space, the data update is more flexible, and the data on HDFS can be directly used without secondary handling, thereby improving the efficiency of data processing.

[0133] The external table is associated with the storage location of the message customer data in the distributed file system.​

[0134] Step S710, load the message customer data in the distributed file system into an external table.

[0135] Specifically, the message customer data stored in HDFS is linked with the Hive external table, enabling Hive to read and process these data without physical movement of data.

[0136] Step S720, identify abnormal data in the message customer data in the external table and clean the abnormal data.

[0137] Specifically, data cleaning is an indispensable step in data preprocessing, aiming to identify and remove outliers such as missing values, error values, etc. in the data, to ensure the accuracy of subsequent data analysis and processing. Data cleaning can significantly improve the accuracy and reliability of data analysis, avoiding the influence of abnormal data on subsequent processing results, thereby improving the accuracy of enterprise decision-making.

[0138] Step S730, convert the cleaned message customer data into a preset data format.

[0139] Specifically, the purpose of data conversion is to convert the original data into a format suitable for data analysis and processing, such as converting from JSON to CSV, or adjusting the field order and name for subsequent processing.

[0140] Step S740, standardize the message customer data in the preset data format.

[0141] Specifically, standardization is a method of unifying the numerical values of data fields within a certain range, eliminating dimensional differences, and comparing and analyzing data of different fields on the same scale, which is particularly important for machine learning model training. The unification and standardization of data format can make data flow more smoothly in different processing links, avoid processing delays caused by incompatible data formats, and simplify the complexity of data processing, improving efficiency.

[0142] In this embodiment, the use of external table and HDFS linkage realizes efficient storage and management of data, avoids multiple data handling, saves storage space and processing time. By identifying and removing abnormal data, the accuracy and reliability of data analysis are guaranteed, avoiding the negative impact of outliers on data analysis results. Data conversion to a preset format makes data processing more flexible, simplifies the data processing process, and improves the efficiency of data flow. Standardization eliminates dimensional differences, improves the consistency of data analysis and model training, and helps improve the accuracy of data analysis and the performance of model prediction.

[0143] Exemplarily, as Figure 8As shown, first, the message customer data collection module collects data from various channels (APP messages, SMS messages, public number messages, etc.) and inputs it into the Hadoop cluster. The Hadoop cluster, as the core of data storage and calculation, stores the data on multiple nodes (Node1, Node2, Node3, etc.) and processes the data through Hive Driver and MapReduce. The strategy management platform reads the filtering strategies from the Oracle database and applies these strategies to the data processing module to achieve filtering and conversion of the data. The processed data is output through the output module for interception statistics, intermediate table synchronization, and channel-specific unloading, etc. operations, and finally outputs the message customer push results. The entire process is monitored by the job scheduling management platform to ensure efficient task execution and fault recovery. This system realizes efficient and flexible processing of large-scale message data through distributed architecture and dynamic filtering strategies.

[0144] In one embodiment, the method further comprises:

[0145] Step S800, establishing a user behavior prediction model.

[0146] Specifically, a prediction model is established using historical user behavior data to predict changes in user behavior patterns and preferences in the future.

[0147] Illustratively, past user behavior logs are collected, including but not limited to message click rate, reception time preference, content preference, etc. Time series analysis, recurrent neural networks or long short-term memory networks and other algorithms are used to train a model that can predict recent changes in user behavior. Model training can be done through online learning or periodic offline batch learning to adapt to changing user behavior.

[0148] Step S810, real-time update of filtering strategies and parameters.

[0149] Specifically, the prediction results are used to adjust the filtering strategies and parameters in the data processing flow in real time to optimize the data filtering process and ensure that the information pushed is most in line with the user's immediate interests and needs.

[0150] Illustratively, a strategy adjustment module is designed that can communicate with the prediction model in real time, receive prediction results, and dynamically update filtering strategies accordingly. For example, if the prediction model predicts that the user's interest in a certain type of information will rise significantly in a certain period of time, the strategy adjustment module can relax the filtering conditions for that type of information in advance and increase the push frequency. At the same time, the module also needs to adjust the data block size, replica number and other parameters according to the prediction results to more effectively utilize storage resources.

[0151] Step S820, adaptive data processing and result feedback.

[0152] Specifically, in the data processing process, through the adaptive mechanism, the real-time update of the filtering strategy and the feedback of the processing effect are used to dynamically adjust the flow of data processing, so as to achieve the best performance and user experience.

[0153] For example, an adaptive processing engine is developed, which can monitor the results and efficiency of data processing, such as the conversion rate of filtered message push, processing delay, etc. According to these feedbacks, the adaptive adjustment is made to each link of data processing, such as data collection frequency, data preprocessing algorithm, weight of filtering strategy, etc., so as to achieve the optimal data processing effect. At the same time, the engine also needs to be closely integrated with the strategy management platform, to reflect the latest state of data processing in real time, and provide basis for further optimization of the strategy.

[0154] In this embodiment, first, the change of user behavior pattern is predicted, which provides basis for subsequent adaptive adjustment, and ensures the foresight and scientificity of strategy adjustment. The prediction result guides the real-time adjustment of filtering strategy and data processing parameters, which not only optimizes the efficiency of data processing, but also improves the personalization level of information push, and embodies the rapid response ability of the system to the change of user behavior. The adaptive processing engine continuously optimizes the data processing flow according to the feedback of processing effect, forming a closed-loop optimization mechanism, which ensures the continuous improvement of the system and continuously improves the immediacy of data processing and user experience.

[0155] Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, or network device, etc.) execute the methods of various embodiments of the present application.

[0156] In this embodiment, a message data filtering device is also provided, which is used to implement the above embodiments and preferred embodiments, which have been described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware or a combination of software and hardware is also possible and is contemplated.

[0157] Figure 9 is a structural block diagram of an optional message data filtering device according to an embodiment of the present application. AsFigure 9 As shown, comprising:

[0158] The data collection module 901 is configured to collect message audience data from a plurality of different message channels.

[0159] The data storage module 902 is configured to store the collected message audience data in a distributed file system.

[0160] The strategy determination module 903 is configured to determine a type of filter condition required according to a target business requirement, select a filter condition of the corresponding type from a preset condition library according to the type of filter condition required, obtain a first condition set, determine a priority of each filter condition in the first condition set according to a historical execution effect of the filter condition, determine a filter order of the filter conditions according to the priorities of the filter conditions, and combine the filter conditions according to the filter order to obtain a filter strategy.

[0161] The filter module 904 is configured to filter the distributed message audience data according to the filter strategy to obtain target audience data, wherein the target audience data is used for pushing to a target object.

[0162] In an exemplary embodiment, the device is further configured to sort the filter conditions according to the priority order of the filter conditions to determine an initial filter order, determine a data density of different types of message audience data according to a data type and a data volume of the message audience data, wherein the data type includes text, image, and video, respectively simulate expected redundant data generated when each filter condition is used to filter the message audience data according to the data density of each type of message audience data and a filter rule of the filter conditions, determine a consumption amount of computer resources when each filter condition is executed according to the expected redundant data corresponding to each filter condition, and adjust the initial filter order according to the consumption amount of computer resources when each filter condition is executed and an execution dependency relationship between the filter conditions to obtain the filter order of the filter conditions.

[0163] In an exemplary embodiment, the device is further configured to obtain identity information and historical behavior data of the target object, determine a behavior mode and a preference condition of the target object according to the identity information and the historical behavior data of the target object, wherein the preference condition includes an active time period of the target object and an on-off setting for information recommendation, and adjust the filter condition according to the behavior mode and the preference condition of the target object to update the filter strategy.

[0164] In an example embodiment, the apparatus is further configured to: store the collected message audience data in a distributed manner using a distributed file system, including: deploying a distributed storage cluster according to an expected data volume and data processing requirements of the message audience data, wherein the storage cluster includes a plurality of storage nodes; deploying a management component on the plurality of nodes in the storage cluster, wherein the management component is configured to manage metadata and computing resources in the distributed file system; configuring parameter settings for storing data in the distributed file system based on the management component, wherein the parameters include block size, number of replicas, and storage directory structure; dividing the collected message audience data into a plurality of data blocks according to the parameter settings, and replicating the plurality of data blocks to obtain a plurality of replicas; and storing the plurality of data blocks and the plurality of replicas on the plurality of storage nodes in a distributed manner.

[0165] In an example embodiment, the apparatus is further configured to: after storing the plurality of data blocks and the plurality of replicas on the plurality of storage nodes in a distributed manner, the method further includes: establishing a database table using a warehouse data tool corresponding to the distributed file system, wherein the database table is pre-defined with fields, data types, and partitioning strategies; importing the audience data stored in the distributed file system into the database table, and establishing a mapping relationship in the database table, wherein the mapping relationship is configured to indicate a correspondence between a storage location of the audience data and a directory of the audience data in the database table.

[0166] In an example embodiment, the apparatus is further configured to: after storing the collected message audience data in a distributed manner using the distributed file system, the method further includes: establishing an external table using a warehouse data tool of the distributed file system, wherein the external table is associated with a storage location of the message audience data in the distributed file system; loading the message audience data in the distributed file system into the external table; identifying abnormal data in the message audience data in the external table, and cleaning the abnormal data; converting the cleaned message audience data into a preset data format; and performing standardization processing on the message audience data in the preset data format.

[0167] Embodiments of the present application also provide a storage medium including a stored program, wherein the program performs any of the above methods when executed.

[0168] Optionally, in the present embodiment, the storage medium can be configured to store program code for performing the following steps:

[0169] S1, collecting message audience data from a plurality of different message channels.

[0170] S2, storing the collected message audience data in a distributed manner using a distributed file system.

[0171] S3, determine a type of filter condition required according to a target business requirement. Select a filter condition of the type required from a preset condition library to obtain a first condition set. Determine a priority of a plurality of filter conditions in the first condition set according to a historical execution effect of each filter condition in the first condition set. Determine a filter order of the plurality of filter conditions according to the priority of the plurality of filter conditions. Combine the plurality of filter conditions according to the filter order to obtain a filter strategy.

[0172] S4, filter the message customer group data stored in a distributed manner using the filter strategy to obtain target customer group data, wherein the target customer group data is used to push to a target object.

[0173] Embodiments of the present application also provide an electronic device comprising a memory and a processor, the memory storing a computer program, and the processor being configured to execute the computer program to perform the steps in any of the above method embodiments.

[0174] Optionally, the electronic device described above can further comprise a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0175] Optionally, in the present embodiment, the processor can be configured to execute the following steps through the computer program:

[0176] S1, collect message customer group data from a plurality of different message channels.

[0177] S2, store the collected message customer group data in a distributed manner using a distributed file system.

[0178] S3, determine a type of filter condition required according to a target business requirement. Select a filter condition of the type required from a preset condition library to obtain a first condition set. Determine a priority of a plurality of filter conditions in the first condition set according to a historical execution effect of each filter condition in the first condition set. Determine a filter order of the plurality of filter conditions according to the priority of the plurality of filter conditions. Combine the plurality of filter conditions according to the filter order to obtain a filter strategy.

[0179] S4, filter the message customer group data stored in a distributed manner using the filter strategy to obtain target customer group data, wherein the target customer group data is used to push to a target object.

[0180] Optionally, in the embodiment, the storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0181] The embodiment of the present application further provides a computer program product, including a non-volatile computer readable storage medium, the non-volatile computer readable storage medium stores a computer program product, and the computer program is executed by a processor to realize the steps of the method in various embodiments of the present application.

[0182] Optionally, in the embodiment, the computer program can be set to realize the following steps when executed by the processor:

[0183] S1, collecting message customer group data from a plurality of different message channels.

[0184] S2, using a distributed file system to distribute and store the collected message customer group data.

[0185] S3, determining the type of required filtering conditions according to the target business requirements. According to the type of required filtering conditions, selecting the filtering conditions of the corresponding type from the preset condition library to obtain a first condition set. According to the historical execution effect of each filtering condition in the first condition set, determining the priority of the plurality of filtering conditions in the first condition set. According to the priority of the plurality of filtering conditions, determining the filtering order of the plurality of filtering conditions. According to the filtering order, combining the plurality of filtering conditions to obtain a filtering strategy.

[0186] S4, using the filtering strategy to filter the distributed and stored message customer group data to obtain target customer group data, wherein the target customer group data is used to push to target objects.

[0187] Optionally, the specific examples in the embodiment can refer to the examples described in the above embodiments and optional implementation manners, and the embodiment will not be described here.

[0188] It should be apparent to those skilled in the art that the modules or steps of the application described above can be implemented with general computing devices, which can be centralized on a single computing device or distributed on a network of multiple computing devices, and optionally implemented with program codes executable by the computing devices, which can be stored in storage devices and executed by the computing devices, and in some cases, the steps shown or described can be executed in an order different from that shown, or made into individual integrated circuit modules, or made into a single integrated circuit module. Thus, the present application is not limited to any particular combination of hardware and software.

[0189] The preferred embodiments of the present application described above are only used to explain the principles of the present application and should not be used to limit the scope of the present application. Any modification, equivalent replacement, improvement, etc. within the principles of the present application should be included in the scope of the present application.

Claims

1. A method for filtering message data, characterized in that, The method includes: Collect customer data from multiple different messaging channels; The collected message customer group data is stored in a distributed manner using a distributed file system; Determine the type of filtering conditions required based on the target business needs; Based on the type of filtering condition required, select the corresponding type of filtering condition from the preset condition library to obtain the first condition set; Based on the historical execution effect of each filtering condition in the first condition set, the priority of multiple filtering conditions in the first condition set is determined; The filtering order of the multiple filtering conditions is determined based on their priority. The multiple filtering conditions are combined according to the filtering order to obtain a filtering strategy; The filtering strategy described above is used to filter message customer group data in distributed storage to obtain target customer group data, wherein the target customer group data is used to push to the target object.

2. The message data filtering method according to claim 1, characterized in that, Determining the filtering order of the multiple filtering conditions based on their priority includes: The multiple filtering conditions are sorted according to their priority order to determine the initial filtering order; Based on the data type and data volume of the message customer group data, the data density of different types of message customer group data is determined, wherein the data type includes text, image, and video; Based on the data density of each type of message customer group data and the filtering rules of the multiple filtering conditions, the expected redundant data to be generated when filtering the message customer group data using each of the filtering conditions is simulated. Based on the expected redundant data corresponding to each filtering condition, determine the amount of computer resources consumed by each filtering condition during execution. Based on the amount of computer resources consumed by each filtering condition during execution and the execution dependencies between the multiple filtering conditions, the initial filtering order is adjusted to obtain the filtering order of the multiple filtering conditions.

3. The message data filtering method according to claim 1, characterized in that, The method further includes: Obtain the target object's identity information and historical behavior data; The behavioral patterns and preferences of the target object are determined based on the target object's identity information and historical behavior data, wherein the preferences include the target object's active time periods and on / off settings for information recommendation; The filtering conditions are adjusted based on the target object's behavior patterns and preferences, and the filtering strategy is updated accordingly.

4. The message data filtering method according to any one of claims 1-3, characterized in that, The step of using a distributed file system to distribute and store the collected message customer data includes: Based on the expected data volume and data processing requirements of the message customer group data, a distributed storage cluster is deployed, wherein the storage cluster includes multiple storage nodes; A management component is deployed on multiple nodes in the storage cluster, wherein the management component is used to manage metadata and computing resources in the distributed file system; The distributed file system stores data based on the parameter settings configured by the management component, wherein the parameters include data block size, number of replicas, and storage directory structure; The collected message customer data is divided into multiple data blocks according to the parameter settings, and the multiple data blocks are copied to obtain multiple copies; The multiple data blocks and the multiple replicas are distributed and stored on the multiple storage nodes.

5. The message data filtering method according to claim 4, characterized in that, After distributing the plurality of data blocks and the plurality of replicas across the plurality of storage nodes, the method further includes: A database table is created using the warehouse data tool corresponding to the distributed file system, wherein the database table has predefined fields, data types, and partitioning strategies; The customer data stored in the distributed file system is imported into the database table, and a mapping relationship is established in the database table, wherein the mapping relationship is used to indicate the correspondence between the storage location of the customer data and the directory of the customer data in the database table.

6. The message data filtering method according to any one of claims 1-3, characterized in that, After distributing the collected message customer data using a distributed file system, the method further includes: An external table is created using the warehouse data tool of the distributed file system, wherein the external table is associated with the storage location of the message customer group data in the distributed file system; Load the message customer group data in the distributed file system into the external table; Identify and clean up abnormal data in the message customer group data in the external table; The cleaned message customer group data is converted into a preset data format; The message customer group data in the preset data format is standardized.

7. A message data filtering device, characterized in that, The device includes: The data collection module is used to collect customer data from multiple different messaging channels; The data storage module is used to distribute the collected message customer data using a distributed file system; The strategy determination module is used to determine the type of filtering conditions required based on the target business needs; select corresponding filtering conditions from a preset condition library according to the type of filtering conditions required to obtain a first condition set; determine the priority of multiple filtering conditions in the first condition set based on the historical execution effect of each filtering condition in the first condition set; determine the filtering order of the multiple filtering conditions according to the priority of the multiple filtering conditions; and combine the multiple filtering conditions according to the filtering order to obtain a filtering strategy. The filtering module is used to filter the message customer group data in the distributed storage using the filtering strategy to obtain the target customer group data, wherein the target customer group data is used to push to the target object.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 6.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 6 through the computer program.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images