Machine learning-based mobile application big data management method and system

Abstract

The application relates to the technical field of big data management, in particular to a mobile application big data management method and system based on machine learning, which comprises the following steps: acquiring a multi-source heterogeneous data stream and extracting a timestamp and a feature identifier, classifying to form a feature data grouping set; analyzing data segment fluctuation characteristics to generate a fluctuation characteristic data list; adjusting a storage queue order to optimize a storage distribution table; comparing a logical attribution number to link a writing process to generate a logical centralized data list; and associating a device identity identifier with a writing path to construct a data node writing organization structure. The application can reconstruct storage arrangement through a feature identifier and a time sequence, aggregate and associate data segments in combination with a logical attribution relationship, improve writing path organization and data distribution consistency, and enhance cross-device scheduling coordination capability.

Description

Technical Field

[0001] This invention relates to the field of big data management technology, and in particular to a method and system for managing big data in mobile applications based on machine learning. Background Technology

[0002] Big data management technology involves technical solutions for the collection, storage, analysis, and application of massive amounts of data. Its core aspects include data collection mechanisms, multi-source heterogeneous data fusion processing, real-time performance assurance, and scenario adaptability. This technology often achieves efficient data processing and accurate prediction by constructing distributed storage architectures, machine learning models, and intelligent scheduling strategies. Methodologically, it mainly relies on data classification and indexing rules, feature extraction methods, and resource allocation strategies to integrate and optimize data across different scenarios and systems. Traditional big data management methods refer to the centralized processing and analysis of data uploaded from terminal devices using a single central node. This method addresses the archiving and analysis of remotely collected data in a multi-terminal environment. Typically, it involves setting data field structures and using transmission protocols such as HTTP or TCP / IP to send user behavior information, device status data, or environmental awareness data generated by terminals to a central node. The central node then processes the data according to a preset algorithm and generates analysis results as needed.

[0003] Existing technologies rely on a central node to uniformly receive data uploaded from terminals during data management operations, and use fixed rules to perform data classification and processing. In complex scenarios, their ability to fuse and process multi-source heterogeneous data is limited, resulting in insufficient feature extraction from the data stream, decreased analysis accuracy, and difficulty in ensuring real-time performance. Furthermore, the lack of a dynamic adjustment mechanism for scenario adaptation and flexible deployment leads to low collaboration efficiency between logical modules, a lack of targeted scheduling behavior, and potential problems such as data processing delays, insufficient prediction accuracy, and limited system scalability. Summary of the Invention

[0004] To address the technical problems existing in the prior art, this invention provides a method and system for managing big data in mobile applications based on machine learning. The technical solution is as follows:

[0005] A machine learning-based method for managing big data in mobile applications includes the following steps:

[0006] S1: Obtain the multi-source heterogeneous data stream uploaded by the terminal device, extract the timestamp and feature identifier of each data, classify the data with the same feature identifier in chronological order, and obtain the feature data group set;

[0007] S2: Based on the feature data grouping set, extract the continuous data segments corresponding to each feature identifier, analyze the time interval and data volume change trend between segments, screen out the data segment sequence with significant fluctuation characteristics, and obtain a list of fluctuation feature data.

[0008] S3: Based on the fluctuation characteristic data list, locate the storage location index of the corresponding data segment in the distributed storage node, call the storage mapping table to adjust the arrangement order of the data segments in the storage queue, organize the adjusted data segments to the specified storage area, and obtain the optimized storage distribution table;

[0009] S4: Based on the optimized storage distribution table, extract the logical ownership number of each data segment, compare the consistency between the logical ownership number and the storage path, filter out the data segment groups with consistent logical ownership, and connect them in the order of logical ownership number to write the process to obtain a logically centralized data list.

[0010] S5: Based on the logically centralized data list, extract the mapping relationship between data segment number and terminal device identity, extract the writing path and sequence information, associate the logical belonging scope with the actual scheduling behavior, and obtain the data node writing organization structure;

[0011] In steps S2 to S4, a machine learning model is introduced. The machine learning model includes a time series prediction sub-model based on Long Short-Term Memory (LSTM) network and a feature clustering sub-model based on K-Means, which are used to identify the fluctuation patterns and logical affiliation distribution of data segments. The model training samples include timestamp sequences, data volume change vectors and device interaction records. The model outputs a fluctuation probability matrix for subsequent storage optimization.

[0012] The data distribution optimization process employs a gradient descent-based dynamic scheduling algorithm, with the objective function being:

[0013] ;

[0014] in, The current time interval is [time interval]. For predicting time intervals, For feature weights, This is the priority vector for the storage queue. For regularization parameters;

[0015] The logical attribution number is an integer sequence number set {L1, L2, ..., Ln}, numbered according to the logical classification results of the LSTM output layer; the feature identifier is a unique string encoding (e.g., FID_A001); the device identifier is a unique terminal device number (e.g., IMEI or MAC address); and the path chain group is an ordered linked list structure. This is used to describe the order of scheduling paths across devices.

[0016] As a further embodiment of the present invention, the tagged data block set includes a timestamp sequence, a context tag set, and a feature vector distribution; the feature data group set includes a feature identifier sequence, a timestamp sequence, and data volume change features; the fluctuation feature data list includes a fluctuation feature identifier, a time interval change value, and a data volume fluctuation state type; the optimized storage distribution table includes a storage location index, a storage mapping relationship, and a storage queue position; the logical centralized data list includes a logical ownership number, a data fragment grouping structure, and a centralized write sequence; and the data node write organization structure includes a device identity mapping relationship, an actual write path, and a logical ownership association status.

[0017] Furthermore, the data structures are transmitted through a unified data signal flow interface, and the data flow is as follows: feature data group set → fluctuation feature data list → optimized storage distribution table → logically centralized data list → data node written to the organizational structure.

[0018] As a further aspect of the present invention, the fluctuation feature data list refers to a sequence of data segments formed by collecting data segments with significant fluctuation features and outputting them in chronological order after identifying the fluctuation features of the data segments; the distributed storage node refers to a storage area structure used for temporarily storing and sorting data segments, and the order of data segments in the storage queue is controlled by indexes and mapping relationships; wherein, the fluctuation feature identification result is calculated and output by the LSTM model to produce a confidence value. ,when If the segment is identified as a significant fluctuation segment, it will proceed to the subsequent mapping calculation stage.

[0019] As a further aspect of the present invention, the logical attribution range refers to the range of logical attribution numbers corresponding to each data segment in the distributed storage architecture, and the logical storage area to which the data segment belongs is determined by comparing the consistency between the data segment number and the logical attribution number; the actual scheduling behavior refers to the mapping state between the execution order of the data segments in the write path and their logical attribution; wherein, the logical attribution module uses a logical number matrix... Implement the mapping between logical ownership numbers and path chain groups, matrix elements Represents logical number Corresponding path nodes The scheduling relationship.

[0020] As a further aspect of the present invention, the specific steps of S1 are as follows:

[0021] S101: Obtain the multi-source heterogeneous data stream uploaded by the terminal device, extract the timestamp and feature identifier of each data, extract the data segments with the same feature identifier, and arrange the corresponding timestamps into the structure corresponding to the feature identifier in chronological order to obtain the feature data time sequence action set.

[0022] S102: Based on the action sequence under the feature identifier in the time sequence action set of the feature data, extract the time value and number of actions between consecutive actions, extract each data segment sequence according to the operation set area, classify the writing segments of the data segments connected to the actions, and obtain a list of consecutive operation segments.

[0023] S103: Based on the correspondence between each action segment and feature identifier in the continuous operation segment list, the action segments are assigned to the corresponding feature identifiers, and the action content under each feature identifier is output separately to obtain a feature data group set.

[0024] The extracted results are then used as the input dataset for the machine learning model. The format is<timestamp,feature_id,value> Triplet.

[0025] As a further aspect of the present invention, the specific steps of S2 are as follows:

[0026] S201: Based on the feature data group set, extract the continuous data segment items corresponding to each feature identifier, analyze them according to the time order of the data segments, and filter the feature identifier sequences that appear continuously in the time order to obtain a continuous set of feature identifiers;

[0027] S202: Based on the continuous set of feature identifiers, compare the time interval and data volume change status between adjacent data segments, split the feature identifier segments where the interval decreases and the data volume distribution is dense, extract the feature identifier sequence that is consistent with the time difference amplitude of the preceding and following data segments, and obtain the data segment behavior trend group.

[0028] S203: Based on the data segment behavior trend group, analyze the degree of overlap between the data volume distribution trend and the frequency of instruction occurrence, identify the feature identifier set in which data segments show continuity in the same feature identifier, and obtain a list of fluctuating feature data.

[0029] In step S203, the fluctuation threshold is screened using LSTM prediction results, and similar fluctuation patterns are merged using K-Means clustering results.

[0030] As a further aspect of the present invention, the specific steps of S3 are as follows:

[0031] S301: Based on the feature identifiers in the fluctuation feature data list, extract the corresponding storage location index in the distributed storage node, call the current storage status table to locate the index position of the feature identifier, verify the current order value of each feature identifier in the storage area, and obtain the feature identifier order index sequence.

[0032] S302: Call the sequence value in the sequence index of the feature identifier, extract the mapping segment corresponding to the feature identifier in the storage mapping table, refer to the current mapping relationship of the feature identifier in the queue, transfer the corresponding position of the feature identifier in the original queue to the execution segment, and obtain the storage mapping position set;

[0033] S303: Based on the feature identifier segment mapping result in the storage mapping location set, synchronously change the feature identifier order in the current storage queue, organize the new order after feature identifier mapping to the end of the adjustable execution queue, and obtain the optimized storage distribution table;

[0034] The queue priority function is calculated in real time during the adjustment process.

[0035] ;

[0036] in, The change in time interval, For the fluctuation range of data volume, and These are the weighting coefficients.

[0037] As a further aspect of the present invention, the specific steps of S4 are as follows:

[0038] S401: Based on the feature identifiers in the optimized storage distribution table, retrieve the logical attribution number corresponding to each feature identifier in the distributed storage architecture, extract the feature identifiers item by item according to the logical attribution relationship, and merge the items with the same attribution into the corresponding list with the same logical attribution number to obtain the feature identifier logical attribution sequence set.

[0039] S402: Based on the logical affiliation sequence set of the feature identifiers, filter feature identifier items with consistent logical affiliation numbers, compare the sequential connection between feature identifiers, identify feature identifier groups that are logically associated, and obtain a logical affiliation feature identifier association set.

[0040] S403: Based on the logical attribution feature identifier association set, feature identifiers with the same attribution number are merged into the write path under the same logical attribution channel, the action connection sequence between feature identifiers is connected, and the sequence of feature identifiers under each logical attribution number is extracted to obtain a logical data list.

[0041] And through the logical mapping matrix The correspondence between the feature identifier and the path node is calculated to achieve synchronous updates of the logical number and the path chain group.

[0042] As a further aspect of the present invention, the specific steps of S5 are as follows:

[0043] S501: Based on the logical attribution number corresponding to each group of feature identifiers in the data list of the logical set, call the device identity identifier in the terminal device configuration parameter set, locate the index mapping path between the feature identifier and the device identity identifier, and compare the matching entries between the feature identifier and the device identity identifier to obtain the feature identifier device mapping sequence;

[0044] S502: Based on the feature identifier device mapping sequence, extract the set of physical storage paths corresponding to the feature identifier, filter the write path content in the mapping path, and compare the paths according to the order of the feature identifiers to obtain the feature identifier write path chain group.

[0045] S503: Call the device pointing field of the path item in the path chain group to write the feature identifier, and perform a positional search with the logical ownership number field of the feature identifier to verify the mapping structure connection status between the ownership number and scheduling order of the feature identifier in the path chain, and obtain the data node writing organization structure.

[0046] The final output path chain group data structure is as follows:

[0047] ;

[0048] The message is passed to the system's write end via a message queue mechanism.

[0049] A machine learning-based mobile application big data management system, the system comprising:

[0050] The data extraction module acquires multi-source heterogeneous data streams uploaded by terminal devices, extracts the timestamp and feature identifier of each data, and classifies data with the same feature identifier in chronological order to obtain a feature data group set;

[0051] Based on the feature data group set, the fluctuation identification module extracts the continuous data segments corresponding to each feature identifier, analyzes the time interval and data volume change trend between segments, and filters out the data segment sequence with significant fluctuation characteristics to obtain a fluctuation feature data list.

[0052] Based on the fluctuation characteristic data list, the storage control module locates the storage location index of the corresponding data segment in the distributed storage node, calls the storage mapping table to adjust the arrangement order of the data segments in the storage queue, and organizes the adjusted data segments into the specified storage area to obtain the optimized storage distribution table.

[0053] Based on the optimized storage distribution table, the logical attribution module extracts the logical attribution number of each data segment, compares the consistency between the logical attribution number and the storage path, filters out the data segment groups with consistent logical attribution, and connects them in the writing process according to the order of the logical attribution numbers to obtain a logically centralized data list.

[0054] Based on the logically centralized data list, the structure mapping module extracts the mapping relationship between data segment numbers and terminal device identifiers, extracts writing path and sequence information, associates logical ownership scope with actual scheduling behavior, and obtains the data node writing organization structure.

[0055] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0056] In this invention, the priority write sequence is extracted by feature identification and time order, the data fragment arrangement structure in the storage area is reconstructed, and data fragments with related features are aggregated into continuous paths by combining logical ownership relationships. A centralized write process within the logical ownership is established, and the write distribution status is extracted by the correspondence between device identity identifier and feature identifier. A data organization structure with ownership boundary, path connectivity and scheduling order features is generated, which enhances the local concentration, structural connection capability and coordination between cross-device paths, and improves the organization of write paths and the consistency of data distribution structure. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the method flow provided in an embodiment of the present invention;

[0058] Figure 2 This is a logical block diagram of the feature data grouping set generation process in an embodiment of the present invention;

[0059] Figure 3 This is an analysis flowchart for generating the fluctuation characteristic data list in an embodiment of the present invention;

[0060] Figure 4 This is a schematic diagram of the structure generated by the optimized storage distribution table in an embodiment of the present invention;

[0061] Figure 5 This is a logical architecture diagram of the data list generated in the logical centralization embodiment of the present invention;

[0062] Figure 6 This is a mapping diagram generated by writing data nodes into the organizational structure in an embodiment of the present invention. Detailed Implementation

[0063] This invention provides a method and system for managing big data in mobile applications based on machine learning, and its specific implementation is described in detail with reference to the accompanying drawings. Figure 1As shown, the overall process of this method includes the collaborative work of a data extraction module, a fluctuation identification module, a storage control module, a logical attribution module, and a structure mapping module, ultimately generating data nodes that are written into the organizational structure. The following will describe in detail the specific implementation of each step with reference to the accompanying drawings.

[0064] First, the data extraction module is responsible for acquiring multi-source heterogeneous data streams uploaded by terminal devices and extracting the timestamp and feature identifier for each data item. In practice, the terminal device transmits the data stream to the data extraction module via a network interface. The module parses the timestamp and feature identifier fields in the data stream according to preset rules. These fields typically exist in the data packet as key-value pairs, such as "timestamp:1698765432" and "feature_id:A001". After extraction, the module categorizes data with the same feature identifier according to the order of the timestamps, forming feature data group sets. Figure 2 As shown, this process involves matching timestamp sequences with feature identifier sequences. The timestamp sequence records the time information of each data point, while the feature identifier sequence marks the category attribute of the data. The logic for generating the feature data group set is to arrange data segments with the same feature identifier in chronological order, thereby forming an ordered set of actions for easier subsequent analysis.

[0065] Next, the fluctuation identification module extracts continuous data segments corresponding to each feature identifier based on the feature data group set, and analyzes the time interval and data volume change trend between segments. Specifically, the module first extracts a continuous set of feature identifiers from the feature data group set, that is, it filters out the feature identifier sequences that appear continuously in time sequence. Subsequently, the module compares the time interval and data volume change state between adjacent data segments, and separates feature identifier segments with decreasing time intervals and dense data volume distribution. This process is as follows: Figure 3 As shown, the module identifies data segment sequences with significant fluctuation characteristics by calculating the time difference between adjacent data segments and the magnitude of data volume changes, ultimately generating a fluctuation characteristic data list. This list includes fluctuation characteristic identifiers, time interval changes, and data volume fluctuation state types, providing a basis for subsequent storage optimization.

[0066] In the volatility identification module, a time series prediction model based on an LSTM (Long Short-Term Memory) neural network is used to identify volatility trends. By training this model, we can predict the volatility level of each data segment, thereby determining whether it should be classified as data with significant volatility characteristics. During model training, the input data includes timestamps, data volume, and historical behavior sequences. The output is the volatility confidence score for each data segment. When the confidence score is higher than a threshold, the data segment is marked as a volatility segment. The loss function of the LSTM model is defined as:

[0067] ;

[0068] in, This represents the actual fluctuation value. Here, n represents the volatility predicted by LSTM, and n is the length of the time series.

[0069] Suppose we have the following actual fluctuation data for a data segment and the predicted value from an LSTM model:

[0070] At time point 1, the actual fluctuation LSTM predicts volatility ;

[0071] At time point 2, the actual fluctuation LSTM predicts volatility ;

[0072] At time point 3, the actual fluctuation LSTM predicts volatility .

[0073] Step 1: Calculate the loss function

[0074] Based on the LSTM loss function formula, calculate the error at each time point and sum them:

[0075] for : ;

[0076] for : ;

[0077] for : ;

[0078] The total loss function is:

[0079] ;

[0080] Step 2: Determine the fluctuation trend

[0081] Based on the comparison between the model's output and the actual fluctuations, if the loss value is less than a certain threshold (e.g., 10), the fluctuation of the data segment is considered acceptable; otherwise, it is judged as significant fluctuation.

[0082] In this example, the loss function The data segment's fluctuation trend is considered normal and not classified as significant fluctuation because it is below the set threshold of 10.

[0083] The storage control module, based on the fluctuation characteristic data list, locates the storage location index of the corresponding data segment in the distributed storage nodes and calls the storage mapping table to adjust the order of the data segments in the storage queue. In specific implementation, the module first extracts the feature identifiers from the fluctuation characteristic data list and retrieves their corresponding storage location indices in the distributed storage nodes. Then, the module calls the current storage status table to verify the priority value of each feature identifier in the storage area, generating a feature identifier priority index sequence. Based on this, the module further calls the storage mapping table to extract the mapping segment corresponding to the feature identifier and, referring to the current mapping relationship of the feature identifier in the queue, transfers its corresponding position in the original queue to the execution segment. For example... Figure 4 As shown, this process involves generating a set of storage mapping locations and synchronously changing the order of storage queues, ultimately resulting in an optimized storage distribution table. The optimized storage distribution table contains information such as storage location indices, storage mapping relationships, and storage queue positions, ensuring a more rational arrangement of data fragments within the storage area.

[0084] In the storage control module, the storage queue optimization employs a gradient descent algorithm to adjust the order of data segments. The optimization objective is to minimize the fluctuation in the storage time interval of data segments. The optimization objective function is:

[0085] ;

[0086] in, It is the timestamp difference of the current segment. It is the predicted time interval. It is the weight of each segment. For regularization parameters, This is the priority vector for the storage queue;

[0087] Assuming there are 4 data segments, the actual time interval and the predicted time interval are as follows:

[0088] Actual time interval: Second, Second, Second, Second;

[0089] Prediction time interval: Second, Second, Second, Second.

[0090] Assume the weight of each data segment is:

[0091] , , , .

[0092] At the same time, set the regularization parameters. .

[0093] Step 1: Calculate the time interval error

[0094] First, calculate the time interval error for each data segment:

[0095] for and : ;

[0096] for and : ;

[0097] for and : ;

[0098] for and : ;

[0099] Step 2: Calculate the weighted time interval error

[0100] Next, we determine the weight of each data segment. Calculate the weighted error:

[0101] for : ;

[0102] for : ;

[0103] for : ;

[0104] for : ;

[0105] Step 3: Calculate the error and sum

[0106] We sum the weighted errors to obtain the total error:

[0107] ;

[0108] Step 4: Calculate the squared 2-norm of the priority vector.

[0109] Suppose there is a storage queue priority vector Its elements represent the storage priority of the data fragment. Here, we simply assume:

[0110] ;

[0111] Calculate the squared 2-norm of the priority vector:

[0112] ;

[0113] Step 5: Calculate the objective function

[0114] Finally, we substitute the time interval error and the squared L2 norm of the priority vector into the optimization objective function formula to obtain the total optimization loss value:

[0115] ;

[0116] The results show that, based on the calculations, the value of the objective function is 13.7. This means that, under the current storage queue order, the optimization loss value is 13.7. Using this loss value to adjust the queue order, this value can be minimized through optimization algorithms (such as gradient descent) to obtain a better data storage order.

[0117] Furthermore, the formula for calculating the storage queue priority function is:

[0118] ;

[0119] in, For the first The time interval between each data segment and the previous segment changes. For the magnitude of change in data volume, and These are the weighting coefficients;

[0120] Suppose we have three data segments with the following timestamps and data volumes:

[0121] The first data segment has a timestamp of 1000 seconds and a data size of 500 bytes;

[0122] The second data segment has a timestamp of 1020 seconds and a data size of 600 bytes;

[0123] The third data segment has a timestamp of 1040 seconds and a data size of 700 bytes.

[0124] Assume the weighting coefficients are set as follows: (Weight of time interval changes) (Weight of changes in data volume);

[0125] Step 1: Calculate the change in time interval ( )

[0126] Second;

[0127] Second;

[0128] Step 2: Calculate the magnitude of change in data volume ( )

[0129] byte;

[0130] byte;

[0131] Step 3: Calculate the priority function

[0132] ;

[0133] ;

[0134] The results show that, according to calculations, data fragments 2 and 3 have the same priority, both 44. This result implies that, in the storage queue, the impact of time interval and data volume variations on priority is balanced (because...). and The values ​​are set to 0.7 and 0.3. If the priority queue uses this priority value to adjust the storage order of data, data fragments 2 and 3 will be further sorted based on other factors such as storage space.

[0135] The logical attribution module extracts the logical attribution number for each data segment based on the optimized storage distribution table and compares the consistency between the logical attribution number and the storage path. The module first retrieves the logical attribution number corresponding to each feature identifier in the distributed storage architecture, and extracts feature identifiers item by item according to the logical attribution relationship. Items with the same attribution are merged into the corresponding list with the same logical attribution number, generating a set of logical attribution sequence sets for feature identifiers. Subsequently, the module filters feature identifier items with the same logical attribution number, compares the sequential connection between feature identifiers, identifies groups of feature identifiers that are logically consecutively associated, and generates a set of logical attribution feature identifier associations. Finally, the module merges feature identifiers with the same attribution number into the write path under the same logical attribution channel, connects the action sequence between feature identifiers, extracts the sequence of feature identifiers under each logical attribution number, and generates a logically set data list. For example... Figure 5 As shown, this process embodies the core link of logical attribution number comparison and write path connection, ensuring a more coherent write process for data fragments within the logical attribution range.

[0136] The structure mapping module extracts the mapping relationship between data segment numbers and terminal device identifiers based on the data list in the logical set, and extracts the write path and sequence information. The module first calls the device identifier in the terminal device configuration parameter set to locate the index mapping path between the feature identifier and the device identifier, generating a feature identifier device mapping sequence. Then, the module extracts the set of physical storage paths corresponding to the feature identifiers, filters the write path content in the mapping path, and compares the paths according to the sequence of feature identifiers to generate a feature identifier write path chain group. Finally, the module calls the device pointer field of the path item in the feature identifier write path chain group and performs a positional search with the logical ownership number field of the feature identifier to verify the mapping structure connection status between the ownership number and scheduling order of the feature identifier appearing in the path chain, generating the data node write organization structure. For example... Figure 6 As shown, this process clarifies the logical association between device identification and actual scheduling behavior, ensuring that data nodes written into the organizational structure have ownership boundaries, path connectivity, and scheduling order characteristics.

[0137] In practical applications, the above method can be applied to large-scale data management scenarios in mobile applications. For example, in a social application, user behavior data (such as likes, comments, and shares) is uploaded to the server via terminal devices. The data extraction module first parses the timestamps and feature identifiers of this behavior data to form a set of feature data groups. The fluctuation identification module analyzes the time intervals and data volume change trends of user behavior data, identifies data segment sequences with significant fluctuation characteristics, and generates a list of fluctuation feature data. The storage control module adjusts the arrangement order of data segments in distributed storage nodes according to the list of fluctuation feature data, generating an optimized storage distribution table. The logical attribution module further compares the consistency between the logical attribution number and the storage path, generating a logically centralized data list. Finally, the structure mapping module extracts the mapping relationship between the data segment number and the terminal device identifier, generates data nodes to be written into the organizational structure, and ensures the coordination of data across device paths.

[0138] Throughout the implementation process, the collaboration between the various modules was crucial. The data extraction module provided foundational data support for subsequent modules; the fluctuation identification module provided a basis for storage optimization by analyzing the fluctuation characteristics of data fragments; the storage control module improved the consistency of the data distribution structure by adjusting the storage queue order; the logical attribution module achieved logical aggregation of data fragments through logical attribution number comparison; and the structure mapping module ensured the integrity of the data node write organization structure by mapping the device identification to the write path. Data flow and logical connections between the modules were achieved through distributed storage nodes, ensuring the efficient operation of the entire system.

[0139] Furthermore, this invention further enhances system performance through specific algorithm design and parameter configuration. For example, in the fluctuation identification module, the calculation of time interval changes uses a sliding window algorithm, while the determination of data fluctuation type is based on a threshold comparison method. In the storage control module, the storage mapping table is updated using hash mapping technology, and the storage queue order is adjusted based on a priority queue algorithm. In the logical attribution module, the consistency comparison of logical attribution numbers uses a binary search algorithm, and the connection of write paths is based on topological sorting technology. In the structure mapping module, the mapping relationship between device identification and write path is implemented through a graph database, and the generation of path chain groups is based on a depth-first search algorithm. The application of these algorithms and technologies ensures the system's efficiency and accuracy when processing large-scale data.

[0140] Through the above specific implementation methods, the present invention achieves efficient management of big data in mobile applications and solves problems such as uneven data distribution, chaotic logical attribution, and insufficient cross-device path coordination in the prior art.

[0141] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for managing big data in mobile applications based on machine learning, characterized in that, Includes the following steps: S1: Obtain the multi-source heterogeneous data stream uploaded by the terminal device, extract the timestamp and feature identifier of each data, classify the data with the same feature identifier in chronological order, and obtain the feature data group set; S2: Based on the feature data grouping set, extract the continuous data segments corresponding to each feature identifier, analyze the time interval and data volume change trend between segments, screen out the data segment sequence with significant fluctuation characteristics, and obtain a list of fluctuation feature data. S3: Based on the fluctuation characteristic data list, locate the storage location index of the corresponding data segment in the distributed storage node, call the storage mapping table to adjust the arrangement order of the data segments in the storage queue, organize the adjusted data segments to the specified storage area, and obtain the optimized storage distribution table; S4: Based on the optimized storage distribution table, extract the logical ownership number of each data segment, compare the consistency between the logical ownership number and the storage path, filter out the data segment groups with consistent logical ownership, and connect them in the order of logical ownership number to write the process to obtain a logically centralized data list. S5: Based on the logically centralized data list, extract the mapping relationship between data segment number and terminal device identity, extract the writing path and sequence information, associate the logical belonging scope with the actual scheduling behavior, and obtain the data node writing organization structure; In steps S2 to S4, a machine learning model is introduced. This model includes a time series prediction sub-model based on Long Short-Term Memory (LSTM) networks and a feature clustering sub-model based on K-Means, used to identify fluctuation patterns and logical attribution distributions of data segments. The model training samples include timestamp sequences, data volume change vectors, and device interaction records. The model outputs a fluctuation probability matrix for subsequent storage optimization. The data distribution optimization process employs a gradient descent-based dynamic scheduling algorithm, with the objective function being: ； in, The current time interval is [time interval]. For predicting time intervals, For feature weights, This is the priority vector for the storage queue. For regularization parameters; The logical attribution number is a set of integer sequence numbers, numbered according to the logical classification results of the LSTM output layer; the feature identifier is a unique string encoding; the device identifier is a unique terminal device number; and the path chain group is an ordered linked list structure used to describe the order of cross-device scheduling paths.

2. The mobile application big data management method based on machine learning according to claim 1, characterized in that, The feature data group set includes feature identifier sequence, timestamp sequence, and data volume change features; the fluctuation feature data list includes fluctuation feature identifier, time interval change value, and data volume fluctuation state type; the optimized storage distribution table includes storage location index, storage mapping relationship, and storage queue position; the logical centralized data list includes logical ownership number, data fragment grouping structure, and centralized write sequence; and the data node write organization structure includes device identity mapping relationship, actual write path, and logical ownership association status. Furthermore, the data structures are transmitted through a unified data signal flow interface, and the data flow is as follows: feature data group set → fluctuation feature data list → optimized storage distribution table → logically centralized data list → data node written to the organizational structure.

3. The mobile application big data management method based on machine learning according to claim 1, characterized in that, The fluctuation feature data list refers to the sequence of data segments formed by collecting data segments with significant fluctuation features and outputting them in chronological order after identifying the fluctuation features of the data segments; the distributed storage node refers to the storage area structure used to temporarily store and sort the data segments, and the order of the data segments in the storage queue is controlled by indexes and mapping relationships; wherein, the fluctuation feature identification result is calculated and output by the LSTM model to produce a confidence value. ,when If the segment is identified as a significant fluctuation segment, it will proceed to the subsequent mapping calculation stage.

4. The mobile application big data management method based on machine learning according to claim 1, characterized in that, The logical attribution range refers to the range of logical attribution numbers corresponding to each data segment in the distributed storage architecture. The logical storage area to which a data segment belongs is determined by comparing the data segment number with the logical attribution number. The actual scheduling behavior refers to the mapping state between the execution order of data segments in the write path and their logical attribution. The logical attribution module uses a logical number matrix... Implement the mapping between logical ownership numbers and path chain groups, matrix elements Represents logical number Corresponding path nodes The scheduling relationship.

5. The mobile application big data management method based on machine learning according to claim 1, characterized in that, The specific steps of S1 are as follows: S101: Obtain the multi-source heterogeneous data stream uploaded by the terminal device, extract the timestamp and feature identifier of each data, extract the data segments with the same feature identifier, and arrange the corresponding timestamps into the structure corresponding to the feature identifier in chronological order to obtain the feature data time sequence action set. S102: Based on the action sequence under the feature identifier in the time sequence action set of the feature data, extract the time value and number of actions between consecutive actions, extract each data segment sequence according to the operation set area, classify the writing segments of the data segments connected to the actions, and obtain a list of consecutive operation segments. S103: Based on the correspondence between each action segment and feature identifier in the continuous operation segment list, the action segments are assigned to the corresponding feature identifiers, and the action content under each feature identifier is output separately to obtain a feature data group set. The extracted results are then used as the input dataset for the machine learning model. The format is<timestamp,feature_id,value> Triplet.

6. The mobile application big data management method based on machine learning according to claim 1, characterized in that, The specific steps of S2 are as follows: S201: Based on the feature data group set, extract the continuous data segment items corresponding to each feature identifier, analyze them according to the time order of the data segments, and filter the feature identifier sequences that appear continuously in the time order to obtain a continuous set of feature identifiers; S202: Based on the continuous set of feature identifiers, compare the time interval and data volume change status between adjacent data segments, split the feature identifier segments where the interval decreases and the data volume distribution is dense, extract the feature identifier sequence that is consistent with the time difference amplitude of the preceding and following data segments, and obtain the data segment behavior trend group. S203: Based on the data segment behavior trend group, analyze the degree of overlap between the data volume distribution trend and the frequency of instruction occurrence, identify the feature identifier set in which data segments show continuity in the same feature identifier, and obtain a list of fluctuating feature data. In step S203, the fluctuation threshold is screened using LSTM prediction results, and similar fluctuation patterns are merged using K-Means clustering results.

7. The mobile application big data management method based on machine learning according to claim 1, characterized in that, The specific steps for S3 are as follows: S301: Based on the feature identifiers in the fluctuation feature data list, extract the corresponding storage location index in the distributed storage node, call the current storage status table to locate the index position of the feature identifier, verify the current order value of each feature identifier in the storage area, and obtain the feature identifier order index sequence. S302: Call the sequence value in the sequence index of the feature identifier, extract the mapping segment corresponding to the feature identifier in the storage mapping table, refer to the current mapping relationship of the feature identifier in the queue, transfer the corresponding position of the feature identifier in the original queue to the execution segment, and obtain the storage mapping position set; S303: Based on the feature identifier segment mapping result in the storage mapping location set, synchronously change the feature identifier order in the current storage queue, organize the new order after feature identifier mapping to the end of the adjustable execution queue, and obtain the optimized storage distribution table; The queue priority function is calculated in real time during the adjustment process. ； in, The change in time interval, For the fluctuation range of data volume, and These are the weighting coefficients.

8. The mobile application big data management method based on machine learning according to claim 1, characterized in that, The specific steps of S4 are as follows: S401: Based on the feature identifiers in the optimized storage distribution table, retrieve the logical attribution number corresponding to each feature identifier in the distributed storage architecture, extract the feature identifiers item by item according to the logical attribution relationship, and merge the items with the same attribution into the corresponding list with the same logical attribution number to obtain the feature identifier logical attribution sequence set. S402: Based on the logical affiliation sequence set of the feature identifiers, filter feature identifier items with consistent logical affiliation numbers, compare the sequential connection between feature identifiers, identify feature identifier groups that are logically associated, and obtain a logical affiliation feature identifier association set. S403: Based on the logical attribution feature identifier association set, feature identifiers with the same attribution number are merged into the write path under the same logical attribution channel, the action connection sequence between feature identifiers is connected, and the sequence of feature identifiers under each logical attribution number is extracted to obtain a logical data list. And through the logical mapping matrix The correspondence between the feature identifier and the path node is calculated to achieve synchronous updates of the logical number and the path chain group.

9. The mobile application big data management method based on machine learning according to claim 1, characterized in that, The specific steps of S5 are as follows: S501: Based on the logical attribution number corresponding to each group of feature identifiers in the data list of the logical set, call the device identity identifier in the terminal device configuration parameter set, locate the index mapping path between the feature identifier and the device identity identifier, and compare the matching entries between the feature identifier and the device identity identifier to obtain the feature identifier device mapping sequence; S502: Based on the feature identifier device mapping sequence, extract the set of physical storage paths corresponding to the feature identifier, filter the write path content in the mapping path, and compare the paths according to the order of the feature identifiers to obtain the feature identifier write path chain group. S503: Call the device pointing field of the path item in the path chain group to write the feature identifier, and perform a positional search with the logical ownership number field of the feature identifier to verify the mapping structure connection status between the ownership number and scheduling order of the feature identifier in the path chain, and obtain the data node writing organization structure. The final output path chain group data structure is as follows: ； The message is passed to the system's write end via a message queue mechanism.

10. A mobile application big data management system based on machine learning, characterized in that, The system is used to implement the machine learning-based mobile application big data management method according to any one of claims 1 to 9, the system comprising: The data extraction module acquires multi-source heterogeneous data streams uploaded by terminal devices, extracts the timestamp and feature identifier of each data, and classifies data with the same feature identifier in chronological order to obtain a feature data group set; Based on the feature data group set, the fluctuation identification module extracts the continuous data segments corresponding to each feature identifier, analyzes the time interval and data volume change trend between segments, and filters out the data segment sequence with significant fluctuation characteristics to obtain a fluctuation feature data list. Based on the fluctuation characteristic data list, the storage control module locates the storage location index of the corresponding data segment in the distributed storage node, calls the storage mapping table to adjust the arrangement order of the data segments in the storage queue, and organizes the adjusted data segments into the specified storage area to obtain the optimized storage distribution table. Based on the optimized storage distribution table, the logical attribution module extracts the logical attribution number of each data segment, compares the consistency between the logical attribution number and the storage path, filters out the data segment groups with consistent logical attribution, and connects them in the writing process according to the order of the logical attribution numbers to obtain a logically centralized data list. Based on the logically centralized data list, the structure mapping module extracts the mapping relationship between data segment numbers and terminal device identifiers, extracts writing path and sequence information, associates logical ownership scope with actual scheduling behavior, and obtains the data node writing organization structure.

Images