Big data processing method and system based on user behavior information

Abstract

The invention relates to the Internet information technology, and provides a big data processing method and system based on user behavior information.The method comprises the steps that a burying point SDK is implanted in a client side, behavior data of a user are collected, a data cleaning node analyzes original data of the client side, the analyzed behavior data are put into a log file, and the log file is stored in a database; the data processing node performs format verification and conversion on the behavior data to form converted data, and the data screening node monitors and calculates the converted data in the message queue through the self-calculation factor to form a data trend chart to be displayed in the client. Compared with the prior art, the online user number of the client can be monitored in real time on the data operation platform, operation personnel can conveniently adjust the operation popularization strategy in time, and precise operation is achieved.

Description

Technical Field

[0001] The present invention relates to the field of Internet information technology, in particular to data analysis and operation of news clients in the media industry, and to a method for identifying illegal operation behaviors of business system data based on artificial intelligence. Background Art

[0002] In today's digital information age, the way news is disseminated and obtained has undergone tremendous changes. As a convenient and efficient news dissemination platform, news clients have become an important channel for people to obtain news information. With the rapid development of mobile Internet and the widespread popularity of smart phones, the number of users of news clients continues to grow, and market competition is becoming increasingly fierce.

[0003] In order to stand out from the competition, news client providers need to deeply understand user needs, provide personalized news services, and improve user experience and satisfaction. The key to achieving this goal lies in the in-depth analysis of news client user behavior.

[0004] Traditional news dissemination methods mainly rely on the subjective judgment and experience of editors, and cannot accurately understand the real needs and interests of users. With the rise of big data technology, news client providers have begun to use data analysis technology to understand user behavior, optimize news recommendation algorithms, and improve the ability to accurately push news. However, existing user behavior data analysis methods have the following problems: Incomplete data collection: The current news client user behavior data collection is mainly focused on the user's browsing history, click behavior, etc., while the collection of important data such as user reading time, sharing behavior, comment content, etc. is not comprehensive enough, resulting in inaccurate data analysis results.

[0005] Single data analysis method: Existing user behavior data analysis mainly uses statistical analysis methods, which lacks in-depth mining and analysis of data. For example, it is impossible to accurately classify and predict user interest preferences, nor is it possible to gain in-depth insight and understanding of user behavior patterns.

[0006] Lack of personalized services: Although existing news clients provide a certain degree of personalized recommendation services, they often only make simple recommendations based on users' browsing history and interest tags, and cannot truly meet users' personalized needs. For example, it is impossible to make accurate personalized recommendations based on users' reading habits, time preferences, geographic locations, and other factors.

[0007] Inefficient operations: Due to the lack of effective user behavior data analysis and operational management tools, news client providers are inefficient in content planning, recommendation algorithm optimization, user operations, etc., and are unable to respond to market changes and user needs in a timely manner.

[0008] To sum up, in order to solve the problems existing in the existing news client user behavior data analysis methods and improve the user experience and satisfaction of the news client, there is an urgent need for a new news client user behavior data analysis operation system that can comprehensively and accurately collect user behavior data, use advanced data analysis methods for in-depth mining and analysis, provide personalized news services, and improve operational efficiency. Summary of the invention

[0009] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0010] In view of the above-mentioned and / or existing problems in a big data processing method based on user behavior information, the present invention is proposed.

[0011] Therefore, the problem to be solved by the present invention is There are also SaaS platforms on the market that collect and analyze user behavior data. However, the source of the analyzed data can only be the user behavior data of the client. The data source is relatively single, the industry characteristics are not obvious, and customized analysis cannot be performed for specific scenarios. At the same time, these SaaS platforms are basically based on offline computing and analysis, and cannot meet the real-time requirements of operations.

[0012] In order to solve the above technical problems, the present invention provides the following technical solutions: a big data processing method based on user behavior information, which comprises: By implanting the tracking SDK in the client, the user's behavior data is collected and uploaded to the backend server through the HTTP protocol. The user behavior data includes the user's click preview behavior on the client; After receiving the user behavior data of the HTTP protocol, the load balancer of the backend server balances the pressure of each data cleaning node. The data cleaning node parses the original data of the client and puts the parsed behavior data into the log file; The data processing node verifies and converts the behavior data format to form converted data, and writes it into the stored message queue; The data screening node monitors and calculates the conversion data in the message queue through the self-calculation factor, and divides the conversion data into user static data information and user dynamic data information; The user static data information is placed in a data warehouse, the user dynamic data information is read in real time via a data recording node at a fixed time, the recorded user dynamic data information content is placed in the data warehouse in units of recording time, and the user dynamic data information recorded in units of recording time and the user static data information are arranged in units of time in the database; The user's dynamic data information is calculated and predicted through the self-calculated factors of the data screening node to obtain the predicted data information, and the data warehouse information is combined with the predicted data information to draw the user behavior data trend, which is presented in the client in the form of a chart.

[0013] As a preferred solution of the method for processing big data based on user behavior information described in the present invention, wherein: the method of collecting user behavior data by implanting a tracking SDK in the client and uploading it to the backend server via the HTTP protocol includes: The user behavior data includes client startup, exit, and click-through of advertisements, news, live broadcasts, and operational activities; After the client collects the user's behavior data, it is temporarily stored locally on the client. Through a pre-designed timing mechanism, the collected behavior data definition is summarized and reported to the back-end server through the HTTP protocol, and received by the load balancer of the back-end server.

[0014] As a preferred solution of the big data processing method based on user behavior information described in the present invention, wherein: the data cleaning node parses the original data of the client including; IP address, User-Agent information; The User-Agent information includes browser type information, version information, and operating system information.

[0015] As a preferred solution of the method for processing big data based on user behavior information described in the present invention, wherein: the data processing node verifies and converts the behavior data format to form conversion data, including: The data format verification includes breaking down the user behavior data verification into three levels of verification: syntax, semantics, and behavior, each of which is independent but has information interoperability; The verification results are used to generate a reverse training rule model to form a closed-loop optimization. Different types of data are assigned differentiated processing pipelines, and the user behavior data that has passed the verification is converted through the processing pipeline; The reverse training rule generation model includes automatically adjusting the verification strictness according to the timeliness of the data, verifying the spatiotemporal alignment of the client text log and the real-time operation video, encoding the verification rules into quantum bit states, generating unique spatiotemporal codes for each piece of data, and realizing superlinear rule combination.

[0016] As a preferred solution of the big data processing method based on user behavior information described in the present invention, the superlinear rule combination includes: Calculate the comprehensive verification score S of the unique spatiotemporal code and select a processing strategy based on the score: When S≥0.9, the data processing node directly converts the user behavior data and marks it as credible data; When 0.7≤S<0.9, the data processing node writes the user behavior data into a temporary queue for manual review; When S<0.7, the data processing node transfers the user behavior data to the error correction pipeline for interception; The calculation method of the comprehensive verification score S includes: Wherein a is the semantic score, b is the language score, λ(t) identifies the time-sensitive function, the semantic score is obtained by checking the semantic checking layer, and the language score is obtained by checking the language checking layer; The time-sensitive calculation method is: Where t is the time when the data is generated.

[0017] In view of the above-mentioned and / or existing problems in a big data processing method based on user behavior information, the present invention is proposed.

[0018] Therefore, in order to solve the above technical problems, the present invention provides the following technical solutions: a big data processing system based on user behavior information, comprising: A front-end server module 100, a back-end server module 200 and a storage module 300; The front-end server module 100 includes a client input unit 101 and a client display unit 102. The client input unit includes an embedded SDK for collecting data information input by the user in the client input unit 101. The client display unit 102 displays the user static data, predicted user data and user behavior data trend of the savings module 300. The backend server module 200 includes a load balancer unit 201, a data cleaning unit 202, a data processing unit 203 and a data screening unit 204; After receiving the data input by the client input unit 101, the load balancer unit 201 distributes the user behavior data to the data cleaning unit 202 for cleaning after analyzing the load capacity of the data cleaning unit 202. The data processing unit 203 verifies and converts the cleaned user behavior data to form converted data, and writes it into the message queue of the storage module 300. The data screening unit 204 monitors and calculates the conversion data in the message queue of the storage module 300 through the self-calculation factor to form user static data information and user dynamic data information. The data screening unit 204 processes the user dynamic data information to form user data information and predicted data information within a stage, inserts the user data information within the stage into the user static data information in chronological order to form complete user static data information, puts the complete user static data information into the first block of the storage module 300, puts the predicted data information into the second block of the storage module 300, combines the data warehouse information with the predicted data information to draw the user behavior data trend, and presents it in the form of a chart in the third block of the storage module 300; The storage content of the storage module 300 can be displayed on the client display unit 102 .

[0019] As a preferred solution of the big data processing system based on user behavior information described in the present invention, the data cleaning unit 202 cleans the user data information in the following manner: Constructing a user behavior knowledge graph and presenting the user abnormal pattern data in the client input unit 101 for the customer to confirm. If the abnormal data is presented in the client input unit 101 and the customer chooses to confirm and fills in the password, the behavior data information confirmed by the user is submitted to the data processing unit 203. If the abnormal data is presented in the client input unit 101 and the customer does not choose to confirm or does not fill in the correct password, the abnormal data of the user is deleted. Constructing a user behavior knowledge graph includes recording user nodes to establish a knowledge graph, wherein the user nodes include user ID, user device type, and user common functions.

[0020] As a preferred solution of a big data processing system based on user behavior information described in the present invention, the storage content of the storage module 300 can be displayed in the client display unit 102 including a first block, a second block and a third block, and the first block, the second block and the third block are arranged sequentially in the interface display of the client display unit 102.

[0021] The present invention provides the following technical solution: an electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement a big data processing method based on user behavior information.

[0022] The present invention provides the following technical solution: an electronic device, comprising: A computer-readable storage medium stores executable instructions, which, when executed by a processor, enable the processor to implement a big data processing method based on user behavior information.

[0023] As a preferred solution of the big data processing system based on user behavior information described in the present invention, the display module displays the illegal operations after displaying the data input by the classifier module, and performs warning signal processing on the illegal behavior and the location where the illegal behavior data is generated.

[0024] The present invention proposes a data expansion method and system based on a generative adversarial network. Compared with the prior art, the present invention can monitor the number of online users of the client in real time on the data operation platform, so that the operation personnel can adjust the operation and promotion strategy in time and realize precise operation. Through calculation and analysis, users are grouped and feature tags are added, so that the operation personnel can realize personalized recommendations based on user location, interests and other aspects to meet diverse operation needs. At the same time, the operation platform can present statistical data and trend charts of client news, live broadcasts, activities and other businesses, so that decision-makers can more conveniently grasp the operation status of client applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them: Figure 1 This is a flow chart of an implementation method of a big data processing method based on user behavior information in Example 1.

[0026] Figure 2 This is a system structure flow chart of a big data processing system based on user behavior information in Example 2. DETAILED DESCRIPTION

[0027] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0028] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0029] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0030] Example 1 Reference Figure 1 This is the first embodiment of the present invention, which provides a big data processing method based on user behavior information, which includes: Step 1: User behavior data collection like Figure 1 As shown, by implanting the tracking SDK in the client, the user's behavior data is collected, including client startup, exit, advertising, news, live broadcast, and operational activities. After the data is collected, it is temporarily stored locally on the client. Through a pre-designed timing mechanism, the collected behavior data is reported to the backend service in batches through the HTTP protocol, avoiding frequent calls to the data interface to affect the performance of the client and backend services.

[0031] Step 2: User behavior data transfer After the user behavior data is reported via the HTTP protocol, it reaches the backend load balancer. The load balancer is responsible for balancing the pressure of each data processing node. The processing process mainly includes parsing the original information requested by the client, such as IP address, User-Agent (including browser type, version, operating system, etc.). The processed data is recorded in the log file on the local disk, waiting for further processing by Flume. Flume is responsible for reading the behavior data in the log file, and after simple format verification and conversion, it is written into the storage message queue Kafka.

[0032] Step 3: Real-time analysis and processing of user behavior data Real-time data processing is implemented based on the Flink framework. Through self-developed operators, data in the message queue Kafka is monitored in real time, and real-time online user numbers, access volume and other data are calculated according to business needs. The calculated result data is stored in Redis cache, and the cached data is updated to MySQL in a scheduled task. In addition, the operator is also responsible for inserting the processed behavior data into ElasticSearch in batches for detailed query and backtracking of subsequent business systems.

[0033] In the inductive example, the core creativity is reflected in the dynamic coupling of layered processing and feedback loops: Layered abstraction: Data verification is broken down into three levels: syntax, semantics, and behavior. Each layer is independent but information is interconnected. Feedback drive: Verification results are used to reversely train rules to generate models, forming a closed-loop optimization; Heterogeneous parallelism: Different types of data are assigned differentiated processing pipelines to break through the bottleneck of a single pipeline.

[0034] Deepen into a multi-dimensional verification architecture with overlapping time and space, with specific innovations including: Dynamic weight allocation: automatically adjust the verification strictness according to the timeliness of data; Cross-modal correlation verification: spatiotemporal alignment verification of text logs and real-time operation videos; Quantum rule engine: Encode the verification rules into quantum bit states to achieve superlinear rule combination.

[0035] Specific implementation architecture 1. Heterogeneous Data Pipeline Construction Real-time streaming pipeline (<50ms latency): Assign FPGA-accelerated syntax check units, use convolutional neural networks (CNNs) to detect log format anomalies in real time, and process 100,000 logs per second.

[0036] Batch pipeline (>1TB / batch): Deploy a distributed semantic analysis cluster to identify logical contradictions across log files through knowledge graph association verification.

[0037] Mixing pipe (dynamic load): When the data flow rate exceeds the threshold, part of the data is automatically routed to the edge node for preprocessing, and the core node focuses on complex verification.

[0038] 2. Quantization Rules Engine Encoding traditional rules (such as regular expressions) into quantum states: The α and β weights are dynamically adjusted based on the historical verification accuracy; Constructing a quantum entangled check pair: The log timestamp field is entangled with the NTP time signal of the user operation video. If the verification of either side fails, a cross-modal alarm will be triggered immediately.

[0039] 3. Time-space overlap verification Longitudinal verification (time dimension): A Markov chain model is established to predict the probability distribution of log sequences and intercept abnormal sequences that deviate from the 3σ interval.

[0040] Horizontal verification (spatial dimension): The association between cross-server logs is stored on the blockchain, and distributed consistency is verified using zero-knowledge proof.

[0041] 4. Dynamic Weight Allocation Mechanism Define the time sensitivity coefficient: (t is the time when data is generated, unit: minute) Check strictness formula: When S<0.7, the data is triggered to be reprocessed.

[0042] 5. Closed-loop optimization system The verification results are input into the reinforcement learning model to generate new quantization rules; Perform dimension compression analysis every week to remove redundant rules with contribution < 5%; The adversarial generative network is used to simulate attack scenarios and continuously train the anomaly detection model.

[0043] Implementation Process 1. Data Ingestion and Pipeline Distribution When a log file enters the system, it is quickly classified using the leading 128-byte feature vector; Automatically assign processing pipelines based on data type (e.g., audit logs / performance logs) and urgency; Generate a unique time and space code (Timestamp⊕ServerIP⊕LogType) for each piece of data.

[0044] 2. Quantization parallel verification Real-time streaming pipeline execution: CNN syntax detection (detecting character set anomalies, field missing, etc.), quantum state rule matching (verifying 1,000 regular expressions at the same time); Batch pipeline execution: Knowledge graph association analysis (such as time series verification of user login logs and permission change logs); Byzantine fault tolerance verification across server logs.

[0045] 3. Cross-modal association verification Extract the operation video of the same period from the video storage system; Verify using the multimodal alignment algorithm: The temporal and spatial consistency between the operation commands in the log and the actual operation in the video; Correlation analysis between server performance logs and cabinet infrared thermal imaging.

[0046] 4. Dynamic Decision Making and Routing Calculate the comprehensive verification score S; Select a processing strategy based on the score: S≥0.9: directly write to Kafka and mark as trusted data; 0.7≤S<0.9: Write to the temporary queue and wait for manual review; S<0.7: Go to the error correction pipeline for re-cleaning.

[0047] 5. Ring Optimization Iteration Collect false positive / negative samples from each pipeline; Optimize the rule weight distribution through quantum annealing algorithm; Generate new quantum verification rule sets every week and verify the effects through A / B testing.

[0048] Step 4: Import user behavior data into the data warehouse The user behavior data processed in the previous step is stored in ElasticSearch. To meet the requirements of offline computing, the data needs to be written to a Hive-based data warehouse through Spark.

[0049] Step 5: Import business system data into the data warehouse To solve the problem of incomplete data collection, offline computing requires not only user behavior data but also business data accumulated by the application system. Business data is mainly used to provide multi-dimensional statistical solutions and information expansion during offline computing, such as article information, channel information, activity information, and the relationship between channel articles. Since business systems have their own independent storage media and data formats, it is impossible to adopt a unified data import solution. Therefore, it is necessary to customize the development of related import programs to extract business data and import it according to the preset format of the data warehouse.

[0050] Step 6: Data analysis and processing This step mainly uses the Spark computing platform to perform offline analysis and calculations on the data in the data warehouse, and finally writes the statistical results into MySQL. For example, the client daily activity statistics task is scheduled at 1:00 a.m. every day to calculate the number of client users, visits, length of stay and other indicators of the previous day. Combined with the session ID of user behavior, the user's behavior can be connected in series to form the user's behavior trajectory, which is different from labeling the user's interests and attributes through rules, so as to operate specific user groups more finely and provide more personalized services and information.

[0051] Step 7: Data analysis results presentation The statistical result data of the previous step is stored in MySQL and presented to the operation personnel on the "Data Operation Platform" by drawing data trend and distribution charts.

[0052] Example 2 Reference Figure 2 , which is a second embodiment of the present invention, a big data processing system based on user behavior information includes: A front-end server module 100, a back-end server module 200 and a storage module 300; The front-end server module 100 includes a client input unit 101 and a client display unit 102. The client input unit includes an embedded SDK for collecting data information input by the user in the client input unit 101. The client display unit 102 displays the user static data, predicted user data and user behavior data trend of the savings module 300. The backend server module 200 includes a load balancer unit 201, a data cleaning unit 202, a data processing unit 203 and a data screening unit 204; After receiving the data input by the client input unit 101, the load balancer unit 201 distributes the user behavior data to the data cleaning unit 202 for cleaning after analyzing the load capacity of the data cleaning unit 202. The data processing unit 203 verifies and converts the cleaned user behavior data to form converted data, and writes it into the message queue of the storage module 300. The data screening unit 204 monitors and calculates the conversion data in the message queue of the storage module 300 through the self-calculation factor to form user static data information and user dynamic data information. The data screening unit 204 processes the user dynamic data information to form user data information and predicted data information within a stage, inserts the user data information within the stage into the user static data information in chronological order to form complete user static data information, puts the complete user static data information into the first block of the storage module 300, puts the predicted data information into the second block of the storage module 300, combines the data warehouse information with the predicted data information to draw the user behavior data trend, and presents it in the form of a chart in the third block of the storage module 300; The storage content of the storage module 300 can be displayed on the client display unit 102 .

[0053] The data cleaning unit 202 cleans the user data information in the following manners: Constructing a user behavior knowledge graph and presenting the user abnormal pattern data in the client input unit 101 for the customer to confirm. If the abnormal data is presented in the client input unit 101 and the customer chooses to confirm and fills in the password, the behavior data information confirmed by the user is submitted to the data processing unit 203. If the abnormal data is presented in the client input unit 101 and the customer does not choose to confirm or does not fill in the correct password, the abnormal data of the user is deleted. Constructing a user behavior knowledge graph includes recording user nodes to establish a knowledge graph, wherein the user nodes include user ID, user device type, and user common functions.

[0054] The storage content of the storage module 300 can be displayed in the client display unit 102 and includes a first block, a second block and a third block. The first block, the second block and the third block are arranged in sequence in the interface display of the client display unit 102 .

[0055] Example 3 A third embodiment of the present invention is a big data processing system based on user behavior information, comprising: 1. Experimental preparation and implementation process In order to verify the innovativeness of the big data processing method of the present invention, a test environment simulating real scenarios was built. The test platform includes 3 groups of client clusters (10,000 simulated devices for Android / iOS / Web), 6 data cleaning node servers (Intel Xeon Gold 6338, 128GB RAM), 2 data processing nodes (AMD EPYC 7763, 1TB NVMeSSD) and a distributed storage system (Ceph cluster, total capacity 500TB). The experimental data comes from 50 million behavior logs generated by simulating 100,000 users for 72 hours, including 12 types of events such as client startup, news clicks, and ad browsing.

[0056] The implementation process strictly follows the technical route of the present invention: Data collection stage: a custom tracking SDK is implanted in the simulation client, and a 1-minute timed reporting mechanism is set. The collection fields include basic metadata such as device ID, timestamp, event type, etc., and the screen coordinate sequence of the operation trajectory is recorded (with a resolution of 0.1%).

[0057] Load balancing optimization: Deploy a dynamic weight distribution algorithm on the backend server to automatically adjust the traffic distribution ratio based on the real-time CPU occupancy rate of each cleaning node (collection frequency 10Hz). When the node load exceeds 70%, horizontal expansion is triggered, and the containerized deployment of new nodes can be completed within 30 seconds.

[0058] Multi-level data verification: Building a three-level verification pipeline of syntax, semantics and behavior: Verify JSON format compliance and field integrity at the syntax level (18 mandatory validation rules) The semantic layer detects abnormal spatiotemporal sequences through pre-trained models (such as cross-regional operations by the same user within 0.1 seconds). The behavioral layer compares the consistency of the client video hash value and log description (using the SHA-3 algorithm) Dynamic data processing: A user portrait cube is built in the data warehouse. Static data (device model, registration information, etc.) is stored in columns, and dynamic data (click stream, dwell time, etc.) is segmented by 5-minute time windows. The prediction model uses an improved LSTM network and introduces an attention mechanism to capture behavioral pattern mutation points.

[0059] Visual feedback: Implement multi-dimensional data drilling function in the front-end management interface, and support the generation of heat maps and trend curves based on combination conditions such as user groups, time granularity (1 minute to 24 hours), and event types.

[0060] 2. Experimental data table Table 1: Load balancing performance comparison (response time unit: ms) Table 2: Comparison of data verification effects (unit: %) Table 3: Data processing timeliness (unit: ms) Table 4: Behavior prediction accuracy (unit: %) 3. Data Analysis and Conclusion Experimental data show that the present invention has significant advantages in the entire data processing chain: Improved load balancing efficiency: As shown in Table 1, in a scenario with 100,000 concurrent users, the dynamic weight algorithm reduces the response time from the timeout state of the traditional solution to 397ms (P<0.01, t-test), while reducing hardware costs by 15.8%. This is due to the synergy between the timing mechanism and the load sensor, which achieves sub-second dynamic adjustment of traffic distribution by real-time collection of node resource utilization (sampling error <0.5%).

[0061] Data verification accuracy breakthrough: Table 2 shows that the three-level verification system has increased the comprehensive interception rate to 99.9%, an increase of 7.4 percentage points compared to the single-layer verification. In particular, in terms of spatiotemporal contradiction detection, the combination of semantic layer and behavioral layer verification has achieved an accuracy rate of 99.1%, effectively identifying 0.03% of malicious forged data. This verifies the effectiveness of spatiotemporal coding technology - by quantum coding the hash value (SHA3-512) of the client operation video and the spatiotemporal description of the log to achieve tamper-proof verification.

[0062] Optimization of processing timeliness: As shown in Table 3, in the processing of millions of data, the present invention reduces the end-to-end delay from 1832ms in the prior art to 1205ms (a decrease of 34.2%). This is attributed to the application of superlinear rule combination: when the comprehensive verification score S ≥ 0.9 (accounting for 92.7% of the data), it directly enters the high-speed processing channel to avoid redundant calculations. The parallel parsing capability of spatiotemporal coding increases the cleaning speed by 1.8 times.

[0063] Prediction model innovation: Table 4 shows that the improved LSTM+Attention model has an average improvement of 25.4% in key indicators. By analyzing the time correlation of user dynamic data, the model can automatically identify behavioral chain patterns such as "evening news click → next day ad conversion". The prediction delay is controlled within 100ms to meet the real-time recommendation needs, thanks to the 5-minute granularity optimization of the dynamic data window (comparative experiments show that the 1-minute granularity prediction accuracy is only improved by 1.2%, but the delay increases by 3 times).

[0064] The above data confirm that the present invention surpasses existing technical solutions in core indicators such as data processing efficiency, system stability, and prediction accuracy, and provides a new technical paradigm for large-scale user behavior analysis.

[0065] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the present invention.

Claims

1. A big data processing method based on user behavior information, characterized in that: The method comprises: By implanting the tracking SDK in the client, the user's behavior data is collected and uploaded to the backend server through the HTTP protocol. The user behavior data includes the user's click preview behavior on the client; After receiving the user behavior data of the HTTP protocol, the load balancer of the backend server balances the pressure of each data cleaning node. The data cleaning node parses the original data of the client and puts the parsed behavior data into the log file; The data processing node verifies and converts the behavior data format to form converted data, and writes it into the stored message queue; The data screening node monitors and calculates the conversion data in the message queue through the self-calculation factor, and divides the conversion data into user static data information and user dynamic data information; The user static data information is placed in a data warehouse, the user dynamic data information is read in real time via a data recording node at a fixed time, the recorded user dynamic data information content is placed in the data warehouse in units of recording time, and the user dynamic data information recorded in units of recording time and the user static data information are arranged in units of time in the database; The user's dynamic data information is calculated and predicted through the self-calculated factors of the data screening node to obtain the predicted data information, and the data warehouse information is combined with the predicted data information to draw the user behavior data trend, which is presented in the client in the form of a chart.

2. A method for processing big data based on user behavior information as claimed in claim 1, characterized in that: The process of collecting user behavior data by implanting a tracking SDK in the client and uploading it to the backend server via the HTTP protocol includes: The user behavior data includes client startup, exit, and click-through of advertisements, news, live broadcasts, and operational activities; After the client collects the user's behavior data, it is temporarily stored locally on the client. Through a pre-designed timing mechanism, the collected behavior data definition is summarized and reported to the back-end server through the HTTP protocol, and received by the load balancer of the back-end server.

3. A method for processing big data based on user behavior information as claimed in claim 2, characterized in that: The data cleaning node analyzes the original data of the client including: IP address, User-Agent information; The User-Agent information includes browser type information, version information, and operating system information.

4. A method for processing big data based on user behavior information as claimed in claim 3, characterized in that: The data processing node verifies and converts the behavior data format to form conversion data, including: The data format verification includes breaking down the user behavior data verification into three levels of verification: syntax, semantics, and behavior, each of which is independent but has information interoperability; The verification results are used to generate a reverse training rule model to form a closed-loop optimization. Different types of data are assigned differentiated processing pipelines, and the user behavior data that has passed the verification is converted through the processing pipeline; The reverse training rule generation model includes automatically adjusting the verification strictness according to the timeliness of the data, verifying the spatiotemporal alignment of the client text log and the real-time operation video, encoding the verification rules into quantum bit states, generating unique spatiotemporal codes for each piece of data, and realizing superlinear rule combination.

5. A method for processing big data based on user behavior information as claimed in any one of claim 4, characterized in that: The superlinear rule combination includes: Calculate the comprehensive verification score S of the unique spatiotemporal code and select a processing strategy based on the score: When S≥0.9, the data processing node directly converts the user behavior data and marks it as credible data; When 0.7≤S<0.9, the data processing node writes the user behavior data into a temporary queue for manual review; When S<0.7, the data processing node transfers the user behavior data to the error correction pipeline for interception; The calculation method of the comprehensive verification score S includes: Wherein a is the semantic score, b is the language score, λ(t) identifies the time-sensitive function, the semantic score is obtained by checking the semantic checking layer, and the language score is obtained by checking the language checking layer; The time-sensitive calculation method is: Where t is the time when the data is generated.

6. A big data processing system based on user behavior information, characterized in that: The system comprises a front-end server module (100), a back-end server module (200) and a storage module (300); The front-end server module (100) comprises a client input unit (101) and a client display unit (102), wherein the client input unit contains an embedded SDK for collecting data information input by a user in the client input unit (101), and the client display unit (102) displays the user static data, predicted user data and user behavior data trend of the savings module (300); The backend server module (200) comprises a load balancer unit (201), a data cleaning unit (202), a data processing unit (203) and a data screening unit (204); After receiving the data input by the client input unit (101), the load balancer unit (201) distributes the user behavior data to the data cleaning unit (202) for cleaning after analyzing the load capacity of the data cleaning unit (202); the data processing unit (203) verifies and converts the cleaned user behavior data to form converted data, and writes the converted data into the message queue of the storage module (300); The data screening unit (204) monitors and calculates the conversion data in the message queue of the storage module (300) through the self-calculation factor to form user static data information and user dynamic data information. The data screening unit (204) processes the user dynamic data information to form user data information and predicted data information within a stage. The user data information within the stage is inserted into the user static data information in chronological order to form complete user static data information. The complete user static data information is placed in the first block of the storage module (300), the predicted data information is placed in the second block of the storage module (300), the data warehouse information is combined with the predicted data information to draw a trend of user behavior data, and the trend is presented in the form of a chart in the third block of the storage module (300); The storage content of the storage module (300) can be displayed on the client display unit (102).

7. A big data processing system based on user behavior information as claimed in claim 6, characterized in that: The data cleaning unit (202) cleans the user data information in the following manner: Constructing a user behavior knowledge graph and presenting abnormal user pattern data in a client input unit (101) for confirmation by the client; if the abnormal data is presented in the client input unit (101) and the client chooses to confirm and fills in the password, submitting the user confirmed behavior data information to a data processing unit (203); if the abnormal data is presented in the client input unit (101) and the client does not choose to confirm or does not fill in the correct password, deleting the abnormal user data; The construction of the user behavior knowledge graph includes recording user nodes to establish a knowledge graph, and the user nodes include user ID, user device type and user common functions.

8. A big data processing system based on user behavior information as claimed in claim 6, characterized in that: The storage content of the storage module (300) can be displayed in the client display unit (102) and includes a first block, a second block and a third block, and the first block, the second block and the third block are arranged in sequence in the interface display of the client display unit (102).

9. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the method according to any one of claims 1 to 5.

Images