Clue data mining method and device based on Flink and multimode matching and medium

Abstract

The invention discloses a clue data mining method and device based on Flink and multimode matching and a medium, and the method comprises the steps: obtaining to-be-processed data, carrying out the analysis and packaging of the to-be-processed data, obtaining a structured data package, and transmitting the structured data package to a Kafka message queue; obtaining full-amount rule data from the Redis cache, and constructing a plurality of construction trees of matching types according to the full-amount rule data; loading the construction tree and the corresponding matching type into a Caffeine cache; obtaining a structured data packet from the Kafka message queue, determining a target matching type of the structured data packet, and obtaining a target construction tree from the Caffeine cache according to the target matching type; and performing data matching on the structured data packet through the target construction tree to obtain target clue data. The method improves the efficiency and accuracy of clue data mining, reduces the consumption of system resources, and can be widely applied to the technical field of data processing.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a clue data mining method, device and medium based on Flink and multi-mode matching. Background Art

[0002] In the current era of Internet information explosion, the explosive growth of user behavior data has become an irreversible trend. Especially on various Internet platforms, users participate in interactions through comments, likes, sharing, etc. These behavioral data not only reflect the preferences and needs of users, but also constitute an important part of the platform content ecology. However, with the massive accumulation of comment data, traditional data processing methods have gradually exposed performance bottlenecks, low matching efficiency, insufficient accuracy, and excessive resource consumption, which seriously restrict the improvement of user experience and the optimization of platform operation efficiency. The above problems need to be solved urgently.

[0003] Terminology explanation:

[0004] Flink: Apache Flink is an open source stream processing framework mainly used for distributed data stream and batch processing.

[0005] Secondary cache: It is a mechanism used to cache data in a computer system, usually located between the primary cache and the main memory. In databases and distributed systems, the secondary cache is also often used to cache frequently accessed data to reduce the number of visits to the underlying storage system and increase data access speed.

[0006] Multi-mode matching: An algorithm or technique for finding matches in multiple patterns. In fields such as data processing and text analysis, multi-mode matching algorithms are widely used to quickly identify data items that match specific patterns or rules from large amounts of data. Multi-mode matching algorithms are usually efficient, flexible, and scalable, and can handle multiple types of patterns and matching rules.

[0007] Matching data: Comparing and matching data from multiple data sources to find related or consistent data records. In data processing and analysis, matching data is an important step to ensure data accuracy, completeness, and consistency.

[0008] Full data: refers to all the data in a data set or data source, that is, a data set that has not been filtered or sampled. In Redis or other data storage systems, full data usually refers to the sum of all data stored in the system. Summary of the invention

[0009] The purpose of the present invention is to solve one of the technical problems existing in the prior art to at least a certain extent.

[0010] To this end, an object of an embodiment of the present invention is to provide a clue data mining method based on Flink and multi-mode matching, which improves the efficiency and accuracy of clue data mining and reduces the consumption of system resources.

[0011] Another object of an embodiment of the present invention is to provide a clue data mining device based on Flink and multi-mode matching.

[0012] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present invention include:

[0013] On the one hand, an embodiment of the present invention provides a clue data mining method based on Flink and multi-mode matching, comprising the following steps:

[0014] Acquire the data to be processed, parse and encapsulate the data to be processed to obtain a structured data packet, and then send the structured data packet to the Kafka message queue;

[0015] Obtain full rule data from the Redis cache, and construct a construction tree of multiple matching types based on the full rule data;

[0016] Loading the construction tree and the corresponding matching type into a Caffeine cache;

[0017] Obtaining the structured data packet from the Kafka message queue, and determining a target matching type of the structured data packet, and then obtaining a target structure tree from the Caffeine cache according to the target matching type;

[0018] The structured data packet is matched with the target construction tree to obtain target clue data.

[0019] Further, in one embodiment of the present invention, the step of obtaining the data to be processed, parsing and encapsulating the data to be processed, and obtaining a structured data packet specifically includes:

[0020] Acquire the data to be processed from the networking platform in real time based on Flink;

[0021] Parsing the data to be processed according to preset rules to obtain structured data;

[0022] The structured data is encapsulated to obtain the structured data packet.

[0023] Further, in one embodiment of the present invention, the step of obtaining the full amount of rule data from the Redis cache and constructing a plurality of matching type construction trees according to the full amount of rule data specifically includes:

[0024] Obtaining Redis connection information, and obtaining the full amount of rule data from the Redis cache according to the Redis connection information;

[0025] A pattern string matching requirement is obtained, a plurality of matching types are determined according to the pattern string matching requirement, and then a corresponding construction tree is constructed according to the full amount of rule data and the matching type.

[0026] Furthermore, in one embodiment of the present invention, the matching type includes at least one of a prefix matching type, a suffix matching type, a substring matching type and a full text matching type, the prefix matching type is configured to perform a forward traversal and prefix matching on the data through the construction tree, the suffix matching type is configured to perform a reverse traversal and suffix matching on the data through the construction tree, the substring matching type is configured to perform a reverse traversal and substring matching on the data through the construction tree, and the full text matching type is configured to perform a forward traversal and full matching on the data through the construction tree.

[0027] Further, in one embodiment of the present invention, the step of loading the construction tree and the corresponding matching type into a Caffeine cache specifically includes:

[0028] Determine cache parameters according to a preset configuration file, and create the Caffeine cache according to the cache parameters;

[0029] Using the construction tree as the value and the corresponding matching type as the key value, constructing key-value pair data;

[0030] The key-value pair data is loaded into the Caffeine cache.

[0031] Further, in one embodiment of the present invention, determining the target matching type of the structured data packet, and then acquiring the target construction tree from the Caffeine cache according to the target matching type, specifically includes:

[0032] Parsing the structured data packet to obtain the target matching type;

[0033] The target matching type is used as a key value, a corresponding target key-value pair is matched in the Caffeine cache, and then the value of the target key-value pair is used as the target construction tree.

[0034] Further, in one embodiment of the present invention, the step of performing data matching on the structured data packet through the target structure tree to obtain target clue data specifically includes:

[0035] Traversing the structured data packet and reading the data content of the structured data packet through the target construction tree;

[0036] The data content is matched with the pattern string of the target structure tree, and the target clue data is determined according to the matching result.

[0037] On the other hand, an embodiment of the present invention provides a clue data mining device based on Flink and multi-mode matching, including:

[0038] The data encapsulation module is used to obtain the data to be processed, parse and encapsulate the data to be processed, obtain a structured data packet, and then send the structured data packet to the Kafka message queue;

[0039] A structure tree construction module is used to obtain full rule data from the Redis cache and construct structure trees of multiple matching types according to the full rule data;

[0040] A construction tree cache module, used for loading the construction tree and the corresponding matching type into a Caffeine cache;

[0041] A structure tree acquisition module, used to acquire the structured data packet from the Kafka message queue, determine the target matching type of the structured data packet, and then acquire the target structure tree from the Caffeine cache according to the target matching type;

[0042] The data matching module is used to perform data matching on the structured data packet through the target construction tree to obtain target clue data.

[0043] On the other hand, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the clue data mining method based on Flink and multi-mode matching as described above is realized.

[0044] On the other hand, an embodiment of the present invention further provides a storage medium, which is a computer-readable storage medium for computer-readable storage, and stores one or more programs, and the one or more programs can be executed by one or more processors to implement the clue data mining method based on Flink and multi-mode matching as described above.

[0045] The advantages and beneficial effects of the present invention will be partly given in the following description, partly become apparent from the following description, or be understood through the practice of the present invention:

[0046] The embodiment of the present invention obtains the data to be processed, parses and encapsulates the data to be processed, obtains a structured data packet, and then sends the structured data packet to the Kafka message queue, obtains the full amount of rule data from the Redis cache, builds a structure tree of multiple matching types according to the full amount of rule data, loads the structure tree and the corresponding matching type into the Caffeine cache, obtains the structured data packet from the Kafka message queue, and determines the target matching type of the structured data packet, and then obtains the target structure tree from the Caffeine cache according to the target matching type, performs data matching on the structured data packet through the target structure tree, and obtains the target clue data. The embodiment of the present invention introduces the secondary cache mechanism of the Redis cache and the Caffeine cache through Flink, reduces the access frequency to the database, and realizes the accurate screening of the data to be processed by constructing structure trees of different matching types, thereby improving the efficiency and accuracy of clue data mining, and also reducing the consumption of system resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solution in the embodiments of the present invention, the following introduction is made to the drawings required for use in the embodiments of the present invention. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solution of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 A flowchart of a clue data mining method based on Flink and multi-mode matching provided by an embodiment of the present invention;

[0049] Figure 2 A step flow chart of step S101 provided in an embodiment of the present invention;

[0050] Figure 3 A step flow chart of step S102 provided in an embodiment of the present invention;

[0051] Figure 4 A schematic diagram of obtaining full rule data from a Redis cache provided by an embodiment of the present invention;

[0052] Figure 5 A schematic diagram of constructing a construction tree of multiple matching types provided by an embodiment of the present invention;

[0053] Figure 6 A step flow chart of step S103 provided in an embodiment of the present invention;

[0054] Figure 7 A step flow chart of step S104 provided in an embodiment of the present invention;

[0055] Figure 8 A step flow chart of step S105 provided in an embodiment of the present invention;

[0056] Fig. 9 A schematic diagram of data matching based on a secondary cache provided by an embodiment of the present invention;

[0057] Fig.10 A schematic diagram of the overall process of a clue data mining method based on Flink and multi-mode matching provided by an embodiment of the present invention;

[0058] Fig.11 A schematic diagram of the structure of a clue data mining device based on Flink and multi-mode matching provided by an embodiment of the present invention;

[0059] Fig.12 A schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention;

[0060] Fig.13 A schematic diagram of the structure of a storage medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limitations on the present application. It should be noted that, although the functional module division is performed in the system schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order from the module division in the system schematic diagram or the flow chart. For the step numbers in the following embodiments, they are only set for the convenience of explanation, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0062] In the description of the present invention, the meaning of "a plurality" is two or more. If there is a description of the first or the second, it is only for the purpose of distinguishing the technical features, and it cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features. In addition, unless otherwise defined, all technical and scientific terms used in this document have the same meaning as those commonly understood by technicians in the technical field of this application. The terms used in this document are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0063] In today's digital age, big data has become an indispensable part of business operations and user interaction. As user behavior becomes increasingly diverse, the comment information on major platforms has shown an explosive growth trend. These comments, logs and other data are large in quantity, updated quickly, and have low business value density. Enterprise personnel need to grasp customer feedback and business demands in real time, and need to obtain valuable data content from massive data through rule fuzzy matching. As the rules continue to accumulate and increase, processing performance, efficiency and accuracy issues have gradually become prominent, and have become a bottleneck for business development.

[0064] In terms of processing performance: With the continuous accumulation of comment data and the increasing number of clue rules, traditional data processing architectures are often unable to withstand such a huge data load. Database query speeds decrease, processing delays increase, and may even cause system crashes or downtime, seriously affecting user experience.

[0065] In terms of matching efficiency: Quickly locating and extracting key information that users care about from massive comment data is the key to improving user experience. Traditional simple query methods based on databases are incapable and inefficient when faced with complex matching requirements. Especially in scenarios where multi-condition and multi-mode matching is required, direct access to the database is not only time-consuming, but may also increase the system burden due to frequent data access.

[0066] In terms of matching accuracy: The accuracy and relevance of review information extracted from massive data directly affect user decisions. However, due to the large amount of data, wide sources, and diverse formats, traditional matching methods often cannot guarantee the accuracy of matching results. Problems such as mismatching and missing matches often occur, reducing the practical value of the data.

[0067] Resource consumption: In the process of big data processing, resource consumption is another issue that cannot be ignored. A large amount of CPU calculations, memory usage, and disk I / O operations will increase the operating cost and maintenance difficulty of the system. Especially in real-time processing scenarios, efficient use of resources is particularly important.

[0068] In response to the performance problems, matching efficiency problems, matching accuracy problems and resource consumption problems caused by the above-mentioned massive data, the present invention proposes a clue data mining method based on Flink and multi-mode matching. Through technological innovation and algorithm optimization, it effectively solves the many challenges faced by data processing in a big data environment, and provides strong support for improving user experience and optimizing resource allocation.

[0069] The clue data mining method based on Flink and multi-mode matching provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a set-top box, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the clue data mining method based on Flink and multi-mode matching, etc., but is not limited to the above forms.

[0070] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0071] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0072] The present invention aims to solve the problems of inefficiency and hot information concealment in current data processing through the dual innovation of secondary cache and multi-mode matching algorithm. It can not only greatly improve the data processing speed and efficiency, but also has powerful information screening and presentation capabilities, allowing users to quickly capture valuable information.

[0073] like Figure 1 FIG. 1 is a flowchart of a clue data mining method based on Flink and multi-mode matching provided by an embodiment of the present invention, referring to FIG. Figure 1 The embodiment of the present invention provides a clue data mining method based on Flink and multi-mode matching, which specifically includes the following steps:

[0074] S101, obtaining the data to be processed, parsing and encapsulating the data to be processed to obtain a structured data packet, and then sending the structured data packet to the Kafka message queue.

[0075] like Figure 2 FIG. 1 is a flowchart of step S101 provided in an embodiment of the present invention, referring to FIG. Figure 6 As an optional implementation, the data to be processed is obtained, parsed and packaged to obtain a structured data packet, which specifically includes:

[0076] S1011. Obtain the data to be processed from the networking platform in real time based on Flink;

[0077] S1012, parsing the data to be processed according to preset rules to obtain structured data;

[0078] S1013. Encapsulate the structured data to obtain a structured data packet.

[0079] Specifically, the embodiment of the present invention uses Apache Flink as a real-time stream processing framework to capture data from the Internet platform in real time as data to be processed. Flink can process high-throughput data streams and supports event time (EventTime) and window operations, which is very suitable for processing real-time data.

[0080] Design a data parsing module to parse the captured raw data into structured data according to preset rules (such as JSON format) and encapsulate it into a data packet form.

[0081] The parsed data packets are sent to the Kafka message queue for subsequent consumption and processing. Kafka's high throughput and low latency characteristics can ensure efficient data transmission.

[0082] S102: Obtain full rule data from the Redis cache, and construct a construction tree of multiple matching types based on the full rule data.

[0083] like Figure 3 FIG. 1 is a flow chart of step S102 provided in an embodiment of the present invention, referring to FIG. Figure 3 As an optional implementation, the full amount of rule data is obtained from the Redis cache, and a construction tree of multiple matching types is constructed according to the full amount of rule data, which specifically includes:

[0084] S1021. Obtain Redis connection information, and obtain full rule data from the Redis cache according to the Redis connection information;

[0085] S1022: Obtain pattern string matching requirements, determine multiple matching types according to the pattern string matching requirements, and then construct a corresponding construction tree according to the full amount of rule data and the matching types.

[0086] Specifically, Figure 4 The figure shows a schematic diagram of obtaining full rule data from Redis cache provided by an embodiment of the present invention. When the program starts, the Redis connection is first initialized. The specific process is as follows:

[0087] 1) First, load the Redis configuration information, including server address, port, password, etc., and initialize the Redis connection pool;

[0088] 2) Loading full matching data: Through Redis connection, obtain full matching data (such as keywords, user behavior patterns, etc.) from the Redis database. This data will be used for subsequent matching and screening.

[0089] 3) Local data compensation service: Enable local data compensation service to deal with abnormal situations such as Redis service unavailability or network delay. When Redis cannot provide data, backup data can be obtained from the local cache to ensure the continuity and stability of the system.

[0090] After all the rule data in the secondary cache Redis is loaded, in response to the pattern string matching requirements in the data, the embodiment of the present invention subdivides the multi-pattern string matching algorithm into matching strategies of multiple different matching types and constructs a corresponding construction tree.

[0091] like Figure 5 FIG. 1 is a schematic diagram of constructing a construction tree of multiple matching types provided by an embodiment of the present invention, referring to FIG. Figure 5, further as an optional implementation, the matching type includes at least one of a prefix matching type, a suffix matching type, a substring matching type and a full text matching type, the prefix matching type is configured to perform forward traversal and prefix matching on the data through a construction tree, the suffix matching type is configured to perform reverse traversal and suffix matching on the data through a construction tree, the substring matching type is configured to perform reverse traversal and substring matching on the data through a construction tree, and the full text matching type is configured to perform forward traversal and full matching on the data through a construction tree.

[0092] Specifically, the matching types of the construction tree in the embodiment of the present invention include prefix matching type, suffix matching type, substring matching type and full text matching type, which are respectively described as follows:

[0093] 1. Construction tree based on prefix matching:

[0094] 1) After the data is entered, the search is performed in a forward traversal mode, that is, moving forward from the starting position of the data;

[0095] 2) Read the data content character by character through the tree structure, and perform pattern recognition using an automaton built based on a set of pattern strings;

[0096] 3) Calculate the longest common prefix between the suffix of the currently read text and the prefix of a pattern string in the construction tree to optimize the matching process.

[0097] 2. Construction tree based on suffix matching:

[0098] 1) In contrast to prefix matching, this architecture uses a reverse traversal strategy, that is, searching forward from the end of the data;

[0099] 2) After the data is entered, the text position is adjusted by predicting the next possible position of the suffix, thereby reducing unnecessary scanning of the entire text and improving matching efficiency.

[0100] 3. Construction tree based on substring matching (i.e. fuzzy matching):

[0101] 1) Also use reverse traversal, but focus on searching for potential substring matches within the prefix of the shortest member (min(len(pi))) in the pattern string set;

[0102] 2) After the data is entered, the text position jump is determined by quickly identifying these key substrings, which effectively reduces the need for a complete traversal of the entire text sequence and further improves the matching speed.

[0103] 4. Construction tree based on full text matching:

[0104] 1) Follow the traditional forward traversal method and match along the natural flow of data (from beginning to end);

[0105] 2) It relies on a complex automaton structure pre-built on the construction tree to read the data character by character after it is entered and verify its match with the pattern string in real time;

[0106] 3) This method may involve a comprehensive scan of the text, based on a highly optimized automaton design to ensure accurate recognition of all pattern strings, and is suitable for scenarios that require a complete match.

[0107] S103: Load the construction tree and the corresponding matching type into the Caffeine cache.

[0108] like Figure 6 FIG. 1 is a flow chart of step S103 provided in an embodiment of the present invention, referring to FIG. Figure 6 As an optional implementation, the construction tree and the corresponding matching type are loaded into the Caffeine cache, which specifically includes:

[0109] S1031, determining cache parameters according to a preset configuration file, and creating a Caffeine cache according to the cache parameters;

[0110] S1032, using the constructed tree as the value and the corresponding matching type as the key value to construct key-value pair data;

[0111] S1033. Load the key-value pair data into the Caffeine cache.

[0112] Specifically, in order to reduce the database burden, speed up data access, and ensure that the system can still run smoothly under high load, the embodiment of the present invention adds a first-level cache caffeine to ensure the overall data processing performance. The specific process is as follows:

[0113] 1) Initialization of the first-level cache (Caffeine): Configure the caffeine parameters from the configuration file, use Caffeine's newBuilder() method to set the maximum size of the cache, set the data expiration time ttl and other parameters, and finally use the build() method to create a cache instance.

[0114] 2) The constructed matching data tree is put into the cache: After the Caffeine cache is built, different matching data trees are loaded into the Caffeine cache in the local memory. Caffeine is a high-performance Java cache library that provides fast access and flexible expiration policies.

[0115] S104, obtaining a structured data packet from the Kafka message queue, determining a target matching type of the structured data packet, and then obtaining a target construction tree from the Caffeine cache according to the target matching type.

[0116] like Figure 7 FIG. 1 is a flow chart of step S104 provided in an embodiment of the present invention, referring to FIG. Figure 7 As an optional implementation, the target matching type of the structured data packet is determined, and then the target structure tree is obtained from the Caffeine cache according to the target matching type, which specifically includes:

[0117] S1041, parsing the structured data packet to obtain a target matching type;

[0118] S1042: Use the target matching type as the key value, match the corresponding target key-value pair in the Caffeine cache, and then use the value of the target key-value pair as the target construction tree.

[0119] Specifically, the target matching type of the structured data packet is parsed, key-value pair matching is performed in the corresponding caffeine memory according to the target matching type, and finally the value of the target key-value pair is returned as the target construction tree.

[0120] S105, performing data matching on the structured data packet through the target construction tree to obtain target clue data.

[0121] like Figure 8 FIG. 1 is a flow chart of step S105 provided in an embodiment of the present invention, referring to FIG. Figure 8 As an optional implementation method, data matching is performed on the structured data packet through the target construction tree to obtain target clue data, which specifically includes:

[0122] S1051, traversing the structured data packet, and reading the data content of the structured data packet through the target construction tree;

[0123] S1052, matching the data content with the pattern string of the target structure tree, and determining the target clue data according to the matching result.

[0124] Specifically, according to the matching type of the target construction tree, the corresponding data matching method is used to match the data content with keywords and user behavior patterns to obtain the target clue data.

[0125] like Fig. 9The figure shows a schematic diagram of data matching based on the second-level cache provided by an embodiment of the present invention. In some optional embodiments, data matching is first performed through the first-level cache Caffeine. If the cache hits, the matching result is returned. If the cache does not hit, data matching is performed through the second-level cache Redis, and then the matching result is saved to the first-level cache Caffeine.

[0126] The above is a description of the method flow of the embodiment of the present invention. The following is a further description of the implementation process of the embodiment of the present invention in conjunction with a specific embodiment.

[0127] like Fig.10 The figure shows an overall flow diagram of a clue data mining method based on Flink and multi-mode matching provided by an embodiment of the present invention. The real-time multi-mode matching clue data mining process of the embodiment of the present invention is as follows:

[0128] 1) Real-time data source acquisition: Capture data from the Internet platform in real time, parse the captured data into structured data according to predetermined rules, encapsulate it into data packets, and send the parsed data packets to the Kafka message queue.

[0129] 2) Initialize the redis cache mechanism: Get the redis connection parameters configured in the configuration file, connect to the Redis instance that stores the full amount of rule data, and read the full amount of rule data from redis according to the set key.

[0130] 3) Pattern matching tree construction: Based on the rule data obtained from redis, one rule data corresponds to a construction tree, and then the pattern matching tree is constructed.

[0131] 4) Initialize the first-level cache (caffeine): Get the configuration parameters of caffeine in the configuration file, configure the cache size, data expiration time, cache strategy and other parameters. The key at this time is the construction tree type of the matching rule, and the value is the previously constructed pattern matching tree.

[0132] 5) Data processing results: After all initialization is completed, the data set in the Kafka consumer queue is parsed, and the value of the pattern matching type "pattern_type" in the data set is obtained. The value of this pattern matching type corresponds to the key of the construction tree stored in the cache. The matching type is matched with the corresponding construction tree type, and the pattern string stored in the caffeine cache is obtained. The "data" data set in the data set is parsed and matched with the construction tree, and the final result data is returned.

[0133] The embodiment of the present invention adopts a real-time, highly optimized clue rule hit strategy, which deeply integrates Flink's stream processing capabilities, Redis's high-performance caching mechanism, and Caffeine's extremely fast memory caching characteristics, and cleverly integrates advanced pattern matching algorithms. Through this innovative secondary cache architecture design, this solution significantly accelerates the data processing process and greatly improves processing efficiency and response speed. This mechanism not only intelligently caches the core matching links in data processing, effectively reduces the burden on the database, and reduces the need for frequent access to the database, but also relies on the precise computing power of the multi-pattern matching algorithm to achieve in-depth analysis of data content and accurate screening without omissions.

[0134] The embodiment of the present invention introduces a secondary cache mechanism of Redis cache and Caffeine cache through Flink, which significantly improves the speed and efficiency of data processing. This mechanism can not only effectively cache the key matching links in the data processing process and reduce the frequency of access to the database, but also achieve in-depth analysis and precise screening of comment content by constructing structural trees of different matching types. It can intelligently identify and extract comments that have both product uniqueness and user emotional tendencies, and present users with a more refined, efficient and valuable overview of the comment area, thereby improving the efficiency and accuracy of clue data mining and reducing the consumption of system resources.

[0135] Compared with the prior art, the embodiments of the present invention also have the following advantages:

[0136] 1) Optimize the performance of massive data processing: Based on Flink, a secondary cache mechanism is introduced to cache frequently accessed hot data and key matching results in memory or faster storage media, reducing the number of direct accesses to the underlying database and utilizing distributed processing capabilities, thereby significantly reducing the database load pressure and improving the overall processing capability and response time of the system. This cache strategy can not only effectively alleviate the performance bottleneck caused by large data volumes, but also ensure the stable operation of the system in high-concurrency scenarios.

[0137] 2) Improving matching efficiency: The present invention proposes a clue hitting method based on a multi-mode matching algorithm. The algorithm can process multiple matching modes and conditions at the same time, and combined with the distributed processing capability of Flink, it can more effectively improve the matching efficiency, conduct in-depth analysis and mining of comment content, and thus quickly lock in content of interest to users.

[0138] 3) Ensure matching accuracy: Before the matching process, the review data is strictly cleaned and preprocessed, including removing noise, correcting typos, standardizing formats, etc. This helps reduce matching errors caused by data quality issues and improves the accuracy of matching results.

[0139] 4) Optimize resource consumption: The use of a secondary cache mechanism and an efficient data processing algorithm effectively reduces unnecessary resource consumption. On the one hand, the cache mechanism reduces the number of database accesses and reduces disk I / O consumption; on the other hand, the optimized algorithm structure improves computing efficiency and reduces CPU usage time.

[0140] like Fig.11 FIG. 1 is a schematic diagram of a clue data mining device based on Flink and multi-mode matching provided by an embodiment of the present invention, referring to FIG. Fig.11 , an embodiment of the present invention provides a clue data mining device based on Flink and multi-mode matching, including:

[0141] The data encapsulation module is used to obtain the data to be processed, parse and encapsulate the data to be processed, obtain a structured data packet, and then send the structured data packet to the Kafka message queue;

[0142] The structure tree construction module is used to obtain the full amount of rule data from the Redis cache and construct structure trees of multiple matching types based on the full amount of rule data;

[0143] The construction tree cache module is used to load the construction tree and the corresponding matching type into the Caffeine cache;

[0144] The structure tree acquisition module is used to obtain structured data packets from the Kafka message queue, determine the target matching type of the structured data packets, and then obtain the target structure tree from the Caffeine cache according to the target matching type;

[0145] The data matching module is used to match the structured data packet through the target construction tree to obtain the target clue data.

[0146] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0147] The embodiment of the present invention further provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned clue data mining method based on Flink and multi-mode matching. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0148] like Fig.12 FIG. 1 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention, referring to FIG. Fig.12 , an embodiment of the present invention provides an electronic device, including:

[0149] The processor 1201 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention;

[0150] The memory 1202 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1202 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1202, and the processor 1201 calls and executes the clue data mining method based on Flink and multi-mode matching in the embodiment of the present invention;

[0151] Input / output interface 1203, used to implement information input and output;

[0152] The communication interface 1204 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0153] A bus 1205 that transmits information between various components of the device (e.g., the processor 1201, the memory 1202, the input / output interface 1203, and the communication interface 1204);

[0154] The processor 1201 , the memory 1202 , the input / output interface 1203 and the communication interface 1204 are connected to each other in communication within the device via the bus 1205 .

[0155] like Fig.13 FIG. 1 is a schematic diagram of a storage medium according to an embodiment of the present invention. Fig.13 An embodiment of the present invention further provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs 1301, and the one or more programs 1301 can be executed by one or more processors to implement the above-mentioned clue data mining method based on Flink and multi-mode matching.

[0156] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0157] The embodiment of the present invention also discloses a computer program product or a computer program, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes Figure 1 The method shown.

[0158] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the above-mentioned boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0159] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified to the contrary, one or more of the above-mentioned functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0160] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the above methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0161] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0162] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the above-mentioned program is printed, since the above-mentioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or processing in other suitable ways as necessary, and then stored in a computer memory.

[0163] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0164] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0165] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0166] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A clue data mining method based on Flink and multi-mode matching, characterized in that: The following steps are involved: Acquire the data to be processed, parse and encapsulate the data to be processed to obtain a structured data packet, and then send the structured data packet to the Kafka message queue; Obtain full rule data from the Redis cache, and construct a construction tree of multiple matching types based on the full rule data; Loading the construction tree and the corresponding matching type into a Caffeine cache; Obtaining the structured data packet from the Kafka message queue, and determining a target matching type of the structured data packet, and then obtaining a target construction tree from the Caffeine cache according to the target matching type; The structured data packet is matched with the target construction tree to obtain target clue data.

2. According to claim 1, a clue data mining method based on Flink and multi-mode matching is characterized in that: The step of obtaining the data to be processed, parsing and packaging the data to be processed, and obtaining a structured data packet specifically includes: Acquire the data to be processed from the networking platform in real time based on Flink; Parsing the data to be processed according to preset rules to obtain structured data; The structured data is encapsulated to obtain the structured data packet.

3. According to claim 1, a clue data mining method based on Flink and multi-mode matching is characterized in that: The method of obtaining full rule data from the Redis cache and constructing a plurality of matching type construction trees according to the full rule data specifically includes: Obtaining Redis connection information, and obtaining the full amount of rule data from the Redis cache according to the Redis connection information; A pattern string matching requirement is obtained, a plurality of matching types are determined according to the pattern string matching requirement, and then a corresponding construction tree is constructed according to the full amount of rule data and the matching type.

4. According to claim 3, a clue data mining method based on Flink and multi-mode matching is characterized in that: The matching type includes at least one of a prefix matching type, a suffix matching type, a substring matching type and a full text matching type. The prefix matching type is configured to perform a forward traversal and prefix matching on the data through the construction tree, the suffix matching type is configured to perform a reverse traversal and suffix matching on the data through the construction tree, the substring matching type is configured to perform a reverse traversal and substring matching on the data through the construction tree, and the full text matching type is configured to perform a forward traversal and full matching on the data through the construction tree.

5. The clue data mining method based on Flink and multi-mode matching according to claim 1 is characterized in that: The step of loading the construction tree and the corresponding matching type into a Caffeine cache specifically includes: Determine cache parameters according to a preset configuration file, and create the Caffeine cache according to the cache parameters; Using the construction tree as the value and the corresponding matching type as the key value, constructing key-value pair data; The key-value pair data is loaded into the Caffeine cache.

6. The clue data mining method based on Flink and multi-mode matching according to claim 5 is characterized in that: The determining the target matching type of the structured data packet, and then acquiring the target structure tree from the Caffeine cache according to the target matching type, specifically includes: Parsing the structured data packet to obtain the target matching type; The target matching type is used as a key value, a corresponding target key-value pair is matched in the Caffeine cache, and then the value of the target key-value pair is used as the target construction tree.

7. A clue data mining method based on Flink and multi-mode matching according to claims 1 to 6, characterized in that: The step of performing data matching on the structured data packet through the target structure tree to obtain target clue data specifically includes: Traversing the structured data packet and reading the data content of the structured data packet through the target construction tree; The data content is matched with the pattern string of the target structure tree, and the target clue data is determined according to the matching result.

8. A clue data mining device based on Flink and multi-mode matching, characterized in that: include: The data encapsulation module is used to obtain the data to be processed, parse and encapsulate the data to be processed, obtain a structured data packet, and then send the structured data packet to the Kafka message queue; A structure tree construction module is used to obtain full rule data from the Redis cache and construct structure trees of multiple matching types according to the full rule data; A construction tree cache module, used for loading the construction tree and the corresponding matching type into a Caffeine cache; A structure tree acquisition module, used to acquire the structured data packet from the Kafka message queue, determine the target matching type of the structured data packet, and then acquire the target structure tree from the Caffeine cache according to the target matching type; The data matching module is used to perform data matching on the structured data packet through the target construction tree to obtain target clue data.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the clue data mining method based on Flink and multi-mode matching as described in any one of claims 1 to 7 are realized.

10. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the clue data mining method based on Flink and multi-mode matching as described in any one of claims 1 to 7.

Images