Electronic commerce information processing system and method based on big data

Abstract

The invention discloses an electronic commerce information processing system and method based on big data, and relates to the technical field of data processing. The performance data acquisition module is used for acquiring transmission data and a compression algorithm list to compress the transmission data to obtain a compressed packet and performance data; the performance prediction module is used for performing prediction through a performance prediction model to obtain a performance prediction result and determining a parameter optimization strategy; the parameter optimization module is used for carrying out transmission parameter optimization according to the parameter optimization strategy and the performance prediction result to obtain a target transmission parameter and carrying out configuration updating; the data uploading module is used for determining an execution block of each transmission data in the transmission task and performing data compression and uploading; performance data and prediction results are obtained, the system executes a single-task or multi-task optimization strategy, transmission parameters are dynamically adjusted, configuration is updated, data compression and uploading are processed in parallel through execution blocks, the data processing speed and the resource utilization efficiency are improved, and the adaptability and stability of the system in a complex task scene are enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and specifically relates to an e-commerce information processing system and method based on big data. Background Art

[0002] In recent years, with the development of cloud computing and big data technologies, the demand for data transmission of e-commerce information has continued to grow. To meet this challenge, data transmission technology continues to advance, focusing primarily on improving transmission efficiency and reducing costs. For example, data compression technology reduces bandwidth costs by reducing the amount of data transmitted, while content delivery networks reduce latency by caching data close to users. Furthermore, intelligent routing algorithms and edge computing technologies are being used to optimize data transmission paths, further increasing transmission speeds.

[0003] Patent number: CN119377839A, discloses an information processing system based on big data; real-time data processing engine: used to process and analyze real-time data streams to achieve rapid decision-making and response; including distributed computing framework, data acquisition module, data preprocessing module, memory computing module, load balancing module, data storage module, data display module; adaptive machine learning algorithm unit: used to automatically adjust and optimize analysis models to improve the accuracy and efficiency of data processing; including data feature selection, data model establishment and evaluation, online learning and model optimization; multi-layer data security protection unit: used to protect the security of data during transmission and storage; dynamic resource management unit: used to dynamically adjust resource allocation according to system load and resource usage to improve resource utilization efficiency; anomaly detection and early warning unit: used to monitor the system operation status in real time, and promptly detect and respond to anomalies.

[0004] Although some problems have been solved in existing technologies, some still exist. For example, traditional methods rely on virtual machines for data compression and transmission, which has defects such as slow startup and waste of resources. Although serverless computing is flexible, it faces complex parameter configuration, resulting in increased data transmission costs and reduced efficiency. Summary of the Invention

[0005] The purpose of the present invention is to solve the problem that the traditional method relies on virtual machines for data compression and transmission, which has defects such as slow startup and waste of resources. Although serverless computing is flexible, it faces complex parameter configuration, which leads to increased data transmission costs and reduced efficiency. An e-commerce information processing system and method based on big data is proposed.

[0006] In the first aspect of the present invention, an e-commerce information processing system based on big data is first proposed. The system includes: a performance data acquisition module, a performance prediction module, a parameter optimization module, and a data upload module.

[0007] The performance data acquisition module is used to obtain transmission data and a compression algorithm list, and randomly select any compression algorithm to compress the transmission data to obtain a compressed package and performance data; the performance data includes: compression ratio and compression speed; the compression algorithm list includes multiple compression algorithms;

[0008] The performance prediction module is used to input the performance data and transmission data into the performance prediction model to obtain performance prediction results, obtain transmission tasks and determine parameter optimization strategies based on the transmission tasks; the parameter optimization strategies include single-task optimization strategies and multi-task optimization strategies;

[0009] The parameter optimization module is configured to optimize the transmission parameters according to the parameter optimization strategy and the performance prediction result to obtain target transmission parameters, and to update the configuration of the execution block according to the target transmission parameters;

[0010] The data upload module is used to determine the execution block of each transmission data in the transmission task, input each transmission data into the execution block for data compression and uploading; the transmission task contains multiple transmission data.

[0011] Optionally, the system further includes: a historical data acquisition module, a model training module, and a model update module:

[0012] The historical data acquisition module is used to acquire a historical data set and determine the data type of the historical data set, and classify the data in the historical data set according to the data type to obtain multiple historical sub-data;

[0013] The model training module is used to compress the historical sub-data in sequence using various compression algorithms to obtain multiple compressed packages and a historical performance data set, and input the historical performance data and historical sub-data as training data into a preset model for training to obtain model update parameters; the historical performance data set contains multiple historical performance data, and the historical performance data corresponds one-to-one to the compression algorithm;

[0014] The model updating module is used to update the preset model according to the model updating parameters to obtain a performance prediction model; the output of the performance prediction model is the performance data of the unexecuted algorithm.

[0015] Optionally, the performance prediction module includes: a single-task construction module and a single-task optimization module:

[0016] The single-task construction module is used to determine a first constraint condition and construct a single-task transmission model according to the first constraint condition, and randomly generate an initial solution according to the single-task transmission model; the constraint condition includes: time constraint, configuration constraint and parameter constraint;

[0017] The single-task optimization module is used to iteratively optimize the initial solution. When a preset number of iterations is reached, the iterative update is stopped, and the optimal solution is input. The optimal solution is input into the single-task transmission model to obtain the transmission time and cost. If the transmission time and cost meet the constraints, the current configuration is recorded.

[0018] Optionally, the performance prediction module further includes: a multi-task construction module and a multi-task optimization module:

[0019] The multi-task construction module is used to determine a second constraint condition and construct a multi-task transmission model based on the second constraint condition, and generate a dynamic programming array dp(j,k) based on the multi-task transmission model; j represents the first j tasks, and k represents the total concurrency allocated to the j tasks;

[0020] The multi-task optimization module is used to determine the total number of tasks, initialize the dynamic programming array to obtain the initial dynamic programming array dp(0,0), iteratively optimize the initial dynamic programming array dp(0,0) in sequence according to the number of tasks and record the configuration under each number of tasks until the total number of tasks is reached, then stop updating and output the multi-task transmission configuration; the multi-task transmission configuration is the configuration under the total number of tasks.

[0021] Optionally, the parameter optimization module further includes: a bandwidth acquisition module and a bandwidth optimization module:

[0022] The bandwidth acquisition module is used to obtain the actual network bandwidth of any execution block and calculate the deviation rate based on the actual network bandwidth and the preset network bandwidth;

[0023] The bandwidth optimization module is configured to determine that the network bandwidth usage is normal if the first threshold ≤ the deviation rate < the second threshold, and to determine that the network bandwidth usage is abnormal if the second threshold ≤ the deviation rate, and to optimize the network bandwidth parameters of the execution block.

[0024] In a second aspect of the present invention, a method for processing e-commerce information based on big data is proposed, the method comprising:

[0025] Obtaining transmission data and a list of compression algorithms, randomly selecting any compression algorithm to compress the transmission data to obtain a compressed package and performance data; the performance data includes: compression ratio and compression speed; the compression algorithm list includes multiple compression algorithms;

[0026] Inputting the performance data and transmission data into a performance prediction model to obtain a performance prediction result, obtaining a transmission task and determining a parameter optimization strategy based on the transmission task; the parameter optimization strategy includes a single-task optimization strategy and a multi-task optimization strategy;

[0027] Optimize the transmission parameters according to the parameter optimization strategy and the performance prediction result to obtain target transmission parameters, and update the configuration of the execution block according to the target transmission parameters;

[0028] Determine the execution block of each transmission data in the transmission task, input each transmission data into the execution block for data compression and uploading; the transmission task includes multiple transmission data.

[0029] Optionally, before inputting the performance data and transmission data into a performance prediction model to predict and obtain a performance prediction result, the training process of the performance prediction model includes:

[0030] Acquire a historical data set and determine the data type of the historical data set, and classify the data in the historical data set according to the data type to obtain a plurality of historical sub-data;

[0031] Compressing the historical sub-data in sequence using various compression algorithms to obtain multiple compressed packages and a historical performance data set, and inputting the historical performance data and historical sub-data as training data into a preset model for training to obtain model update parameters; the historical performance data set contains multiple historical performance data, and the historical performance data corresponds to the compression algorithm one by one;

[0032] The preset model is updated according to the model update parameters to obtain a performance prediction model; the output of the performance prediction model is performance data of the unexecuted algorithm.

[0033] Optionally, performing transmission parameter optimization according to the parameter optimization strategy and the performance prediction result to obtain target transmission parameters, when the parameter optimization strategy is a single-task optimization strategy, includes:

[0034] Determining a first constraint condition and constructing a single-task transmission model based on the first constraint condition, and randomly generating an initial solution based on the single-task transmission model; the constraint condition includes: time constraint, configuration constraint, and parameter constraint;

[0035] The initial solution is iteratively optimized. When the preset number of iterations is reached, the iterative update is stopped, and the optimal solution is input. The optimal solution is input into the single-task transmission model to obtain the transmission time and cost. If the transmission time and cost meet the constraints, the current configuration is recorded.

[0036] Optionally, performing transmission parameter optimization according to the parameter optimization strategy and the performance prediction result to obtain target transmission parameters, when the parameter optimization strategy is a multi-task optimization strategy, includes:

[0037] Determine a second constraint condition and construct a multi-task transmission model based on the second constraint condition, and generate a dynamic programming array dp(j,k) based on the multi-task transmission model, where j represents the first j tasks and k represents the total concurrency allocated to the j tasks;

[0038] Determine the total number of tasks, initialize the dynamic programming array to obtain an initial dynamic programming array dp(0,0), iteratively optimize the initial dynamic programming array dp(0,0) according to the number of tasks, and record the configuration under each number of tasks until the total number of tasks is reached. Then stop updating and output a multi-task transmission configuration; the multi-task transmission configuration is the configuration under the total number of tasks.

[0039] Optionally, after updating the configuration of the execution block according to the target transmission parameter, the method further includes:

[0040] Obtain the actual network bandwidth of any execution block and calculate the deviation rate based on the actual network bandwidth and the preset network bandwidth;

[0041] If the first threshold ≤ deviation rate < second threshold, it is determined that the network bandwidth usage is normal; if the second threshold ≤ deviation rate, it is determined that the network bandwidth usage is abnormal, and the network bandwidth parameters of the execution block are optimized.

[0042] Beneficial effects of the present invention:

[0043] The present invention proposes an e-commerce information processing method based on big data, which uses a performance data acquisition module to obtain transmission data and a compression algorithm list to compress the transmission data to obtain a compressed package and performance data; a performance prediction module is used to predict through a performance prediction model to obtain performance prediction results and determine a parameter optimization strategy; a parameter optimization module is used to optimize transmission parameters according to the parameter optimization strategy and performance prediction results to obtain target transmission parameters and perform configuration updates; a data upload module is used to determine the execution blocks of each transmission data in the transmission task and perform data compression and upload; after obtaining performance data and prediction results, the system executes a single-task or multi-task optimization strategy, dynamically adjusts transmission parameters and updates configurations, and uses execution blocks to parallelly process data compression and upload, thereby improving data processing speed and resource utilization efficiency, and enhancing the adaptability and stability of the system in complex task scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The present invention will be further described below with reference to the accompanying drawings.

[0045] Figure 1 A framework diagram of an e-commerce information processing system based on big data provided by an embodiment of the present invention;

[0046] Figure 2 A flowchart of an e-commerce information processing method based on big data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments represent only a portion of the embodiments of the present invention, not all of them. The term "and / or" herein simply describes an association relationship between associated objects, indicating that three possible relationships exist. For example, "A" and "B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, references to "first," "second," and so on in the present invention are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include at least one of these features. Furthermore, the technical solutions of the various embodiments may be combined, but only if they are achievable by a person of ordinary skill in the art. If a combination of technical solutions contradicts or is unachievable, such combination shall be deemed non-existent and outside the scope of protection claimed by the present invention.

[0048] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0049] Based on the same inventive concept, the present invention also provides an e-commerce information processing system based on big data. Figure 1 , Figure 1 The framework diagram of the big data-based e-commerce information processing system provided by an embodiment of the present invention includes: a performance data acquisition module, a performance prediction module, a parameter optimization module, and a data upload module:

[0050] The performance data acquisition module is used to obtain the transmission data and the compression algorithm list, and randomly select any compression algorithm to compress the transmission data to obtain a compressed package and performance data;

[0051] The performance prediction module is used to input performance data and transmission data into the performance prediction model to obtain performance prediction results, obtain transmission tasks and determine parameter optimization strategies based on the transmission tasks;

[0052] The parameter optimization module is used to optimize the transmission parameters according to the parameter optimization strategy and performance prediction results to obtain the target transmission parameters, and to update the configuration of the execution block according to the target transmission parameters;

[0053] The data upload module is used to determine the execution block of each transmission data in the transmission task, input each transmission data into the execution block for data compression and upload;

[0054] Performance data includes: compression ratio and compression speed; the compression algorithm list contains multiple compression algorithms; the parameter optimization strategy includes single-task optimization strategy and multi-task optimization strategy; the transmission task contains multiple transmission data.

[0055] The big data-based e-commerce information processing system provided by the embodiment of the present invention obtains performance data and prediction results, executes single-task or multi-task optimization strategies, dynamically adjusts transmission parameters and updates configurations, and utilizes execution blocks to parallel process data compression and upload, thereby improving data processing speed and resource utilization efficiency, and enhancing the system's adaptability and stability in complex task scenarios.

[0056] In one embodiment, the system further includes: a historical data acquisition module, a model training module, and a model update module:

[0057] A historical data acquisition module is used to acquire a historical data set and determine the data type of the historical data set, and classify the data in the historical data set according to the data type to obtain multiple historical sub-data;

[0058] The model training module is used to compress the historical sub-data in sequence using various compression algorithms to obtain multiple compressed packages and a historical performance data set, and input the historical performance data and historical sub-data as training data into a preset model for training to obtain model update parameters; the historical performance data set contains multiple historical performance data, and the historical performance data corresponds one-to-one to the compression algorithm;

[0059] The model update module is used to update the preset model according to the model update parameters to obtain a performance prediction model; the output of the performance prediction model is the performance data of the unexecuted algorithm.

[0060] In one embodiment, the performance prediction module includes: a single-task construction module and a single-task optimization module:

[0061] A single-task construction module is used to determine a first constraint condition and construct a single-task transmission model based on the first constraint condition, and randomly generate an initial solution based on the single-task transmission model; the constraint condition includes: time constraint, configuration constraint and parameter constraint;

[0062] The single-task optimization module is used to iteratively optimize the initial solution. When the preset number of iterations is reached, the iterative update is stopped and the optimal solution is input. The optimal solution is input into the single-task transmission model to obtain the transmission time and cost. If the transmission time and cost meet the constraints, the current configuration is recorded.

[0063] In one embodiment, the performance prediction module further includes: a multi-task construction module and a multi-task optimization module:

[0064] a multi-task construction module, configured to determine a second constraint condition and construct a multi-task transmission model based on the second constraint condition, and generate a dynamic programming array dp(j,k) based on the multi-task transmission model; j represents the first j tasks, and k represents the total concurrency allocated to the j tasks;

[0065] The multi-task optimization module is used to determine the total number of tasks, initialize the dynamic programming array to obtain the initial dynamic programming array dp(0,0), iteratively optimize the initial dynamic programming array dp(0,0) according to the number of tasks and record the configuration under each number of tasks until the total number of tasks is reached. Then, the update stops and the multi-task transmission configuration is output; the multi-task transmission configuration is the configuration under the total number of tasks.

[0066] In one embodiment, the parameter optimization module further includes: a bandwidth acquisition module and a bandwidth optimization module:

[0067] The bandwidth acquisition module is used to obtain the actual network bandwidth of any execution block and calculate the deviation rate based on the actual network bandwidth and the preset network bandwidth;

[0068] The bandwidth optimization module is used to determine that the network bandwidth usage is normal if the first threshold ≤ deviation rate < second threshold, and to determine that the network bandwidth usage is abnormal if the second threshold ≤ deviation rate, and to optimize the network bandwidth parameters of the execution block.

[0069] The embodiment of the present invention provides an e-commerce information processing method based on big data. Figure 2 , Figure 2 This is a flowchart of a method for processing e-commerce information based on big data provided by an embodiment of the present invention. The method includes the following steps:

[0070] S101, obtaining transmission data and a list of compression algorithms, and randomly selecting any compression algorithm to compress the transmission data to obtain a compressed package and performance data;

[0071] S102, inputting the performance data and transmission data into a performance prediction model to obtain a performance prediction result, obtaining a transmission task and determining a parameter optimization strategy according to the transmission task;

[0072] S103, optimizing transmission parameters according to the parameter optimization strategy and the performance prediction results to obtain target transmission parameters, and updating the configuration of the execution block according to the target transmission parameters;

[0073] S104, determining an execution block for each transmission data in the transmission task, and inputting each transmission data into the execution block for data compression and uploading;

[0074] Performance data includes: compression ratio and compression speed; the compression algorithm list contains multiple compression algorithms; the parameter optimization strategy includes single-task optimization strategy and multi-task optimization strategy; the transmission task contains multiple transmission data.

[0075] Based on the big data-based e-commerce information processing method provided by the embodiment of the present invention, by acquiring performance data and prediction results, the system executes single-task or multi-task optimization strategies, dynamically adjusts transmission parameters and updates configurations, and uses execution blocks to parallel process data compression and upload, thereby improving data processing speed and resource utilization efficiency, and enhancing the adaptability and stability of the system in complex task scenarios.

[0076] In one implementation, e-commerce information is uploaded from each child node to the intermediate node. The intermediate node receives the data, compresses it, and then uploads it to the cloud. The intermediate node compresses the data through each execution block (cloud function). The compression algorithm includes multiple compression algorithms, such as Snappy, Zlib, Bzip2, LZMA, etc. Snappy is a lossless compression algorithm developed by Google, with high-speed compression and decompression as its core advantage. It sacrifices some compression ratio in exchange for performance and is suitable for scenarios with high real-time requirements (such as big data processing and distributed storage systems). Zlib is a general-purpose lossless compression library based on DEFLATE. The algorithm (combining LZ77 and Huffman coding) provides a balance between compression ratio and speed and is widely integrated into various software and file formats. Bzip2 is a high-compression lossless compression algorithm based on the Burrows-Wheeler transform and Huffman coding. It typically has a compression ratio of 4:1 to 6:1 and is suitable for highly redundant data such as text and source code. LZMA (Lempel-Ziv-Markov Chain-Algorithm) is an improved version of the lossless compression algorithm LZ77. It is characterized by an extremely high compression ratio (up to 5:1 to 7:1), but has an extremely slow compression speed (well below 1MB / s) and a moderate decompression speed.

[0077] In one implementation, one compression algorithm is selected from multiple compression algorithms and used to compress the transmitted data, obtaining performance data such as compression ratio and compression speed. This data is then input into a performance prediction model for prediction. This process allows for rapid evaluation of the performance of different compression algorithms on actual data, providing accurate input for subsequent parameter optimization. Based on historical data and machine learning techniques, the performance prediction model can predict the performance of other compression algorithms, avoiding the need for actual compression testing of each algorithm. This indirectly improves the system's response speed and resource utilization (reducing the need for actual testing of each compression algorithm), making the optimization process more efficient and accurate.

[0078] In one implementation, a single-task optimization or multi-task optimization strategy is selected based on the characteristics of the transmission task. Single-task optimization focuses on optimizing the parameters of a single task, ensuring that costs are minimized within given time constraints. Multi-task optimization, on the other hand, considers the situation where multiple tasks share resources (i.e., multiple transmission tasks) and minimizes overall costs by jointly optimizing the parameters of multiple tasks. This flexible strategy selection mechanism enables the system to dynamically adjust optimization objectives based on different application scenarios, improving the system's adaptability and flexibility. Through parameter optimization, the system can improve resource utilization efficiency within limited resources.

[0079] In one implementation, based on performance prediction results and the selected parameter optimization strategy, the transmission parameters are optimized to obtain the target transmission parameters, and the execution block is configured and updated accordingly. By dynamically adjusting the memory size, concurrency, data block size and other parameters of the execution block, it is ensured that each task or task group can achieve optimal performance during execution. The dynamic configuration update mechanism can improve the flexibility and response speed of the system, and adapt to different task requirements by adjusting parameters in real time, further improving the overall performance and resource utilization efficiency of the system. The optimized parameter configuration can also reduce data transmission time.

[0080] In one implementation, each transmission data is input into an execution block for data compression and upload. As the basic unit of data processing, the execution block can process multiple data blocks in parallel, improving the speed and efficiency of data processing. By performing data compression and upload within the execution block, the system can fully utilize the elastic resources of serverless computing to achieve efficient data transmission. The parallel processing capabilities of the execution block enable the system to handle multiple transmission tasks simultaneously, further improving system throughput and responsiveness.

[0081] In one implementation, in multitasking scenarios, the system jointly optimizes the parameters of multiple tasks to minimize overall costs while sharing resources. This multitasking mechanism not only improves resource utilization efficiency but also ensures that each task is completed within the specified timeframe by dynamically adjusting task priorities and resource allocation. Furthermore, through precise resource management and task scheduling, the system can better cope with complex task demands, improving system stability and reliability. This comprehensive resource management and task scheduling mechanism enhances the system's overall performance and adaptability.

[0082] In one embodiment, before inputting the performance data and the transmission data into the performance prediction model to obtain the performance prediction results, the training process of the performance prediction model includes:

[0083] Acquire a historical data set and determine the data type of the historical data set, and classify the data in the historical data set according to the data type to obtain multiple historical sub-data;

[0084] The historical sub-data is compressed in sequence by various compression algorithms to obtain multiple compressed packages and historical performance data sets, and the historical performance data and historical sub-data are input into a preset model as training data for training to obtain model update parameters; the historical performance data set contains multiple historical performance data, and the historical performance data corresponds to the compression algorithm one by one;

[0085] The preset model is updated according to the model update parameters to obtain a performance prediction model; the output of the performance prediction model is the performance data of the unexecuted algorithm.

[0086] In one implementation, the performance data consists of compression ratio and compression speed. In order to use them as inputs for the preset model, the compression ratio and compression speed are taken as vectors (for example:<r1,V1> ) is input into the preset model for training; vectors obtained by different compression methods of the same data are input into the preset model for training, so when only the vector of one compression algorithm is input, the performance of other compression algorithms can be predicted; since the compressed information is usually a continuous floating-point number, the regression model is selected as the basic model (for example: random forest model, a type of machine learning model).

[0087] One implementation involves classifying historical datasets to generate multiple sub-data sets. These sub-data sets are then compressed using a variety of compression algorithms to generate rich training data. This process not only ensures the diversity and representativeness of the training data but also provides comprehensive input features and corresponding performance labels for the machine learning model. This data preprocessing approach significantly improves the model's generalization capabilities, enabling it to more accurately predict performance across different data types and compression algorithms.

[0088] One implementation involves training a pre-set model based on generated training data and updating model parameters to ultimately generate a performance prediction model. This model outputs performance data for unexecuted algorithms, enabling rapid evaluation of the performance of different compression algorithms in real-world applications. This avoids the high computational overhead of actual compression testing for each algorithm. This machine learning-based prediction method improves system responsiveness and resource utilization, making the optimization process more efficient and accurate.

[0089] In one embodiment, transmission parameter optimization is performed based on the parameter optimization strategy and the performance prediction result to obtain target transmission parameters. When the parameter optimization strategy is a single-task optimization strategy, the following steps are included:

[0090] Determine a first constraint condition and construct a single-task transmission model based on the first constraint condition, and randomly generate an initial solution based on the single-task transmission model; the constraint condition includes: time constraint, configuration constraint, and parameter constraint;

[0091] The initial solution is iteratively optimized. When the preset number of iterations is reached, the iterative update is stopped and the optimal solution is input. The optimal solution is input into the single-task transmission model to obtain the transmission time and cost. If the transmission time and cost meet the constraints, the current configuration is recorded.

[0092] In one implementation, the mathematical expression of the single-task transmission model is: min m*p*Time(m,p,s,i)+(D / r i )+(D / s); where m represents the execution block memory size, p represents the execution block concurrency, Time(m,p,s,i) represents the data transmission time, D represents the transmission data size, r i Represents the compression ratio, s represents the execution block size, i represents the compression method type, θ represents the parameter space, and Z represents the set of integers; constraints: 1. Time constraint Time(m,p,s,i) ≤ time threshold (ensuring that the data transmission time does not exceed the preset time limit); 2. Configuration constraints s ≤ D / p and s ≤ m (the transmission data size s cannot exceed the amount of data processed by each execution block, and the memory of the execution block must exceed the transmission data size); parameter constraints m,p,s,i∈θ and m,p,i∈Z (parameters m,p,s,i must be in the parameter space θ, and the values of m,p,i are integers).

[0093] In one implementation, a single-task transmission model was constructed and an initial solution was generated by specifying time, configuration, and parameter constraints. This precise constraint modeling approach enables the system to quickly locate a feasible solution within a complex parameter space. The generation of this initial solution provides a starting point for subsequent optimization, avoiding the computational overhead of blind search. In terms of computational efficiency, this approach reduces inefficient computation during the optimization process, speeds up the algorithm's convergence, and thus improves overall computational efficiency.

[0094] In one implementation, the initial solution is iteratively optimized (via a metaheuristic algorithm, such as a genetic algorithm or particle swarm optimization algorithm) until a predetermined number of iterations is reached. This process ensures that the optimal configuration that satisfies the constraints is found within limited computing resources by gradually approaching the optimal solution (i.e., m, p, s, i). This iterative optimization method adjusts the search direction at each iteration based on the performance feedback of the current solution, avoiding the significant computational burden of global search methods (such as grid search). This significantly reduces unnecessary computation and improves the efficiency of parameter search, enabling the system to find high-quality solutions in a shorter time.

[0095] In one implementation, after each iteration, the optimal solution is input into a single-task transmission model to verify whether its transmission time and cost meet the constraints. This verification step ensures that the solution at each step in the optimization process is feasible, preventing further optimization of invalid solutions. Furthermore, recording the configurations that meet the requirements provides reliable parameter settings for subsequent applications. This verification mechanism reduces backtracking and corrections during the optimization process, improves efficiency and stability, and ensures that the system can efficiently find the configuration that meets the requirements.

[0096] In one embodiment, transmission parameter optimization is performed based on the parameter optimization strategy and the performance prediction result to obtain target transmission parameters. When the parameter optimization strategy is a multi-task optimization strategy, the following steps are included:

[0097] Determine the second constraint condition and build a multi-task transmission model based on the second constraint condition, and generate a dynamic programming array dp(j,k) based on the multi-task transmission model; j represents the first j tasks, and k represents the total concurrency allocated to the j tasks;

[0098] Determine the total number of tasks, initialize the dynamic programming array to obtain the initial dynamic programming array dp(0,0), iteratively optimize the initial dynamic programming array dp(0,0) according to the number of tasks and record the configuration under each number of tasks until the total number of tasks is reached. Then stop updating and output the multi-task transmission configuration; the multi-task transmission configuration is the configuration under the total number of tasks.

[0099] In one implementation, the task data transmission cost model is: n is the total number of tasks, m k is the execution block memory size of task k, p k is the execution block concurrency of task k, s k is the transmission data size of task k, i k is the compression method type of task k; Among them, D k is the data size of task k, Time(m k ,p k ,s k ,i k ) is the data transmission time of task k, r ik Is task k using compression method i k Compression ratio; Second constraint: Time (m k ,p k ,s k ,i k )≤ time threshold k (the transmission time of each task must be completed within the time threshold k), s k ≤D k / p k (Transmission data size s kcannot exceed the amount of data processed by each execution block), s k ≤m k (Execution block memory size m k Internal constraints must be met), (The sum of the execution block concurrency of all tasks cannot exceed the reserved concurrency limit), parameter constraint m k ,p k ,s k ,i k ∈θ and m k ,p k ,i k ∈Z(parameter m k ,p k ,s k ,i k must be within the parameter space θ, m k ,p k ,i k The value of is an integer).

[0100] In one implementation, the initial dynamic programming array dp(0,0) represents the cost of zero when there is no task; assuming there are 3 tasks (n=3) and the total concurrency limit is 5 (P max =5), the total number of tasks is j, when the number of tasks is 0, the cost is also 0; based on this, when the number of tasks is 1, dp(1,k) = min(dp(0,kl) + cost(l,1)), dp(0,kl) is known (because, dp(0,0) = 0); when the number of tasks is 2, dp(2,k) = min(dp(1,kl) + cost(l,2)), dp(1,kl) is calculated based on the optimal solution of the first task; when the number of tasks is 3, dp(3,k) = min(dp(2,kl) + cost(l,3)), dp(2,kl) is calculated based on the optimal solutions of the first two tasks.

[0101] In one implementation, a multi-task transfer model is constructed by determining the second constraint. This process ensures that the parameter configuration of each task not only meets the time constraint but also meets the actual resource constraints. This precise constraint modeling enables the system to quickly locate a feasible solution space in a complex parameter space, avoid invalid parameter combinations, and significantly improve the efficiency and accuracy of the optimization process.

[0102] In one implementation, a dynamic programming array is initialized and iterative optimization is performed sequentially based on the number of tasks, recording the configuration for each task number. This process gradually expands the number of tasks and utilizes the solutions to previously solved subproblems to construct the solution to the current problem, avoiding repeated computations and unnecessary resource waste. The iterative optimization mechanism of dynamic programming not only improves the system's computational efficiency but also, by recording intermediate results, provides complete path information for the final multi-task transmission configuration, ensuring the traceability and accuracy of the optimization results.

[0103] In one implementation, a dynamic programming array is gradually updated to ultimately generate and output a multi-task transmission configuration—the optimal configuration for the total number of tasks. This process ensures that the system finds the parameter configuration with the lowest total monetary cost while satisfying all task time constraints and resource limits. This comprehensive optimization approach not only improves overall system performance but also maximizes resource utilization by dynamically adjusting resource allocation.

[0104] In one embodiment, after updating the configuration of the execution block according to the target transmission parameters, the method further includes:

[0105] Obtain the actual network bandwidth of any execution block and calculate the deviation rate based on the actual network bandwidth and the preset network bandwidth;

[0106] If the first threshold ≤ deviation rate < second threshold, it is determined that the network bandwidth usage is normal; if the second threshold ≤ deviation rate, it is determined that the network bandwidth usage is abnormal, and the network bandwidth parameters of the execution block are optimized.

[0107] In one implementation, if the transmission time exceeds the compression time, it's considered an insufficient bandwidth anomaly; otherwise, it's considered insufficient compression performance. Insufficient bandwidth: Reduce the block size (e.g., from 64MB to 16MB) to reduce the amount of data transmitted at a time and mitigate the impact of network fluctuations. Switch to a high-compression algorithm: Increase the compression ratio and reduce the amount of data transmitted (ensuring the compression time is within the constraints). Insufficient compression: Increase the cloud function memory (e.g., from 512MB to 2048MB) to increase compression speed and shorten compression time.

[0108] In one implementation, the deviation rate between the actual network bandwidth and the preset network bandwidth is calculated ( Where Pc is the deviation rate, b s is the actual network bandwidth, b yBy setting a preset network bandwidth and a clear threshold, the system can accurately determine the network bandwidth usage status. This method not only quickly identifies network bandwidth anomalies, but also further distinguishes whether insufficient bandwidth or insufficient compression performance is the cause. This precise assessment mechanism enables the system to promptly identify potential network issues and avoid data transmission delays caused by bandwidth bottlenecks or inefficient compression, significantly improving system reliability and performance.

[0109] In one implementation, targeted parameter optimization measures are taken based on the network bandwidth status. For example, when bandwidth is insufficient, the amount of data transmitted at a time can be reduced by reducing the block size or switching to a high-compression algorithm, thereby mitigating the impact of network fluctuations. When compression performance is insufficient, increasing the cloud function's memory can increase compression speed and shorten compression time. This flexible optimization strategy not only effectively addresses current performance bottlenecks but also maximizes data transmission efficiency without increasing costs, ensuring efficient system operation under various network conditions.

[0110] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and should not be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. An e-commerce information processing system based on big data, characterized by: The system includes: a performance data acquisition module, a performance prediction module, a parameter optimization module and a data upload module: The performance data acquisition module is used to obtain transmission data and a compression algorithm list, and randomly select any compression algorithm to compress the transmission data to obtain a compressed package and performance data; the performance data includes: compression ratio and compression speed; the compression algorithm list includes multiple compression algorithms; The performance prediction module is used to input the performance data and transmission data into the performance prediction model to obtain performance prediction results, obtain transmission tasks and determine parameter optimization strategies based on the transmission tasks; the parameter optimization strategies include single-task optimization strategies and multi-task optimization strategies; The parameter optimization module is configured to optimize the transmission parameters according to the parameter optimization strategy and the performance prediction result to obtain target transmission parameters, and to update the configuration of the execution block according to the target transmission parameters; The data upload module is used to determine the execution block of each transmission data in the transmission task, input each transmission data into the execution block for data compression and uploading; the transmission task contains multiple transmission data.

2. The e-commerce information processing system based on big data according to claim 1 is characterized in that: The system also includes: a historical data acquisition module, a model training module and a model update module: The historical data acquisition module is used to acquire a historical data set and determine the data type of the historical data set, and classify the data in the historical data set according to the data type to obtain multiple historical sub-data; The model training module is used to compress the historical sub-data in sequence using various compression algorithms to obtain multiple compressed packages and a historical performance data set, and input the historical performance data and historical sub-data as training data into a preset model for training to obtain model update parameters; the historical performance data set contains multiple historical performance data, and the historical performance data corresponds one-to-one to the compression algorithm; The model updating module is used to update the preset model according to the model updating parameters to obtain a performance prediction model; the output of the performance prediction model is the performance data of the unexecuted algorithm.

3. The e-commerce information processing system based on big data according to claim 1, characterized in that: The performance prediction module includes: a single task construction module and a single task optimization module: The single-task construction module is used to determine a first constraint condition and construct a single-task transmission model according to the first constraint condition, and randomly generate an initial solution according to the single-task transmission model; the constraint condition includes: time constraint, configuration constraint and parameter constraint; The single-task optimization module is used to iteratively optimize the initial solution. When a preset number of iterations is reached, the iterative update is stopped, and the optimal solution is input. The optimal solution is input into the single-task transmission model to obtain the transmission time and cost. If the transmission time and cost meet the constraints, the current configuration is recorded.

4. The e-commerce information processing system based on big data according to claim 1, characterized in that: The performance prediction module also includes: a multi-task construction module and a multi-task optimization module: The multi-task construction module is used to determine a second constraint condition and construct a multi-task transmission model based on the second constraint condition, and generate a dynamic programming array dp(j,k) based on the multi-task transmission model; j represents the first j tasks, and k represents the total concurrency allocated to the j tasks; The multi-task optimization module is used to determine the total number of tasks, initialize the dynamic programming array to obtain the initial dynamic programming array dp(0,0), iteratively optimize the initial dynamic programming array dp(0,0) in sequence according to the number of tasks and record the configuration under each number of tasks until the total number of tasks is reached, then stop updating and output the multi-task transmission configuration; the multi-task transmission configuration is the configuration under the total number of tasks.

5. The e-commerce information processing system based on big data according to claim 1 is characterized in that: The parameter optimization module also includes: a bandwidth acquisition module and a bandwidth optimization module: The bandwidth acquisition module is used to obtain the actual network bandwidth of any execution block and calculate the deviation rate based on the actual network bandwidth and the preset network bandwidth; The bandwidth optimization module is configured to determine that the network bandwidth usage is normal if the first threshold ≤ the deviation rate < the second threshold, and to determine that the network bandwidth usage is abnormal if the second threshold ≤ the deviation rate, and to optimize the network bandwidth parameters of the execution block.

6. An e-commerce information processing method based on big data, characterized in that: The method comprises: Obtaining transmission data and a list of compression algorithms, randomly selecting any compression algorithm to compress the transmission data to obtain a compressed package and performance data; the performance data includes: compression ratio and compression speed; the compression algorithm list includes multiple compression algorithms; Inputting the performance data and transmission data into a performance prediction model to obtain a performance prediction result, obtaining a transmission task and determining a parameter optimization strategy based on the transmission task; the parameter optimization strategy includes a single-task optimization strategy and a multi-task optimization strategy; Optimize the transmission parameters according to the parameter optimization strategy and the performance prediction result to obtain target transmission parameters, and update the configuration of the execution block according to the target transmission parameters; Determine the execution block of each transmission data in the transmission task, input each transmission data into the execution block for data compression and uploading; the transmission task includes multiple transmission data.

7. The method for processing e-commerce information based on big data according to claim 6, characterized in that: Before inputting the performance data and transmission data into the performance prediction model to predict and obtain the performance prediction result, the training process of the performance prediction model includes: Acquire a historical data set and determine the data type of the historical data set, and classify the data in the historical data set according to the data type to obtain a plurality of historical sub-data; Compressing the historical sub-data in sequence using various compression algorithms to obtain multiple compressed packages and a historical performance data set, and inputting the historical performance data and historical sub-data as training data into a preset model for training to obtain model update parameters; the historical performance data set contains multiple historical performance data, and the historical performance data corresponds to the compression algorithm one by one; The preset model is updated according to the model update parameters to obtain a performance prediction model; the output of the performance prediction model is performance data of the unexecuted algorithm.

8. The method for processing e-commerce information based on big data according to claim 6, characterized in that: Optimizing transmission parameters according to the parameter optimization strategy and the performance prediction result to obtain target transmission parameters, when the parameter optimization strategy is a single-task optimization strategy, includes: Determining a first constraint condition and constructing a single-task transmission model based on the first constraint condition, and randomly generating an initial solution based on the single-task transmission model; the constraint condition includes: time constraint, configuration constraint, and parameter constraint; The initial solution is iteratively optimized. When the preset number of iterations is reached, the iterative update is stopped, and the optimal solution is input. The optimal solution is input into the single-task transmission model to obtain the transmission time and cost. If the transmission time and cost meet the constraints, the current configuration is recorded.

9. The method for processing e-commerce information based on big data according to claim 6, characterized in that: Optimizing transmission parameters according to the parameter optimization strategy and the performance prediction result to obtain target transmission parameters, when the parameter optimization strategy is a multi-task optimization strategy, includes: Determine a second constraint condition and construct a multi-task transmission model based on the second constraint condition, and generate a dynamic programming array dp(j,k) based on the multi-task transmission model, where j represents the first j tasks and k represents the total concurrency allocated to the j tasks; Determine the total number of tasks, initialize the dynamic programming array to obtain an initial dynamic programming array dp(0,0), iteratively optimize the initial dynamic programming array dp(0,0) according to the number of tasks, and record the configuration under each number of tasks until the total number of tasks is reached. Then stop updating and output a multi-task transmission configuration; the multi-task transmission configuration is the configuration under the total number of tasks.

10. The method for processing e-commerce information based on big data according to claim 6, characterized in that: After updating the configuration of the execution block according to the target transmission parameter, the method further includes: Obtain the actual network bandwidth of any execution block and calculate the deviation rate based on the actual network bandwidth and the preset network bandwidth; If the first threshold ≤ deviation rate < second threshold, it is determined that the network bandwidth usage is normal; if the second threshold ≤ deviation rate, it is determined that the network bandwidth usage is abnormal, and the network bandwidth parameters of the execution block are optimized.

Images