Load balancing-based user packet file parsing method, system, device and medium

Abstract

The application relates to the technical field of file analysis, and discloses a user package file analysis method, system, device and medium based on load balancing, which comprises the following steps: carrying out file splitting on a user package file to obtain a user file list; selecting a target server cluster in a preset backend server cluster according to the user file list; calculating the real-time computing resources of each backend server in the target server cluster, and constructing a file allocation strategy according to the real-time computing resources; carrying out file analysis according to the file allocation strategy, calculating the analysis real-time computing resources after file analysis; and iteratively optimizing the file allocation strategy according to the analysis real-time computing resources until each user file is analyzed to obtain the analysis result of the user package file. According to the application, the user files can be reasonably allocated for file analysis according to the real-time load conditions of the backend servers, so that the computing resources of the backend server cluster can be reasonably utilized, and the stability and accuracy of user file analysis can be effectively improved.

Description

Technical Field

[0001] This invention relates to the field of file parsing technology, and in particular to a method, system, device and medium for parsing user packet files based on load balancing. Background Technology

[0002] As software applications become increasingly complex and user bases expand, the volume of data generated is growing explosively. This puts immense pressure on backend services when processing user data files. Especially when dealing with large user data files, traditional parsing methods reveal a series of significant drawbacks. For example, single-machine processing often leads to excessive load and over-consumption of system resources, resulting in extremely low processing efficiency and potentially even causing system crashes.

[0003] However, existing methods for processing user packet files typically involve a single backend server parsing the file. When the file is too large, server resources (such as CPU, memory, and I / O) are excessively consumed, leading to slow system response or even crashes. At the same time, file splitting and processing are usually not dynamically allocated based on the actual load of the backend servers, failing to fully utilize cluster resources and wasting backend service resources. Moreover, if a server fails, the entire parsing task may fail and need to be restarted, lacking an effective fault tolerance mechanism.

[0004] In conclusion, improving the accuracy of user file parsing has become an urgent problem to be solved. Summary of the Invention

[0005] This invention provides an artificial intelligence-based user packet file parsing method, system, computer device, and medium based on load balancing, to solve the problem of low accuracy in existing file parsing methods.

[0006] Firstly, a user packet file parsing method based on load balancing is provided, including: Obtain the user package file, split the user package file to obtain a list of user files; Based on the user file list, select the target server cluster corresponding to the user file list from the preset backend server cluster; Calculate the real-time computing resources of each backend server in the target server cluster, and allocate the user file list to the target server cluster according to the real-time computing resources to obtain a file allocation strategy; According to the file allocation strategy, the allocated files in the user file list are parsed, and the parsed file list and the real-time parsing computing resources of the backend server are calculated. The file allocation strategy is iteratively optimized based on the real-time computing resources for parsing, until the optimized file allocation strategy parses each user file in the parsing file list to obtain the parsing result of the user package file.

[0007] Secondly, a user packet file parsing system based on load balancing is provided, including: The file splitting module is used to obtain user package files, split the user package files, and obtain a list of user files. The server cluster selection module is used to select the target server cluster corresponding to the user file list from the preset backend server clusters based on the user file list. The file allocation strategy construction module is used to calculate the real-time computing resources of each backend server in the target server cluster, and allocate the user file list to the target server cluster according to the real-time computing resources to obtain the file allocation strategy. The file parsing module is used to parse the allocated files in the user file list according to the file allocation strategy, and to calculate the parsed file list and the real-time parsing computing resources of the backend server. The iterative optimization module is used to iteratively optimize the file allocation strategy based on the real-time computing resources for parsing, until the iteratively optimized file allocation strategy parses each user file in the parsing file list to obtain the parsing result of the user package file.

[0008] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described load-balanced user packet file parsing method.

[0009] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described load-balanced user packet file parsing method.

[0010] The aforementioned solution, implemented using a load-balanced user packet file parsing method, system, computer equipment, and storage medium, effectively improves subsequent packet parsing by splitting user packet files, loading each split file in the user file list sequentially, and parsing different split files simultaneously. Selecting a target server cluster based on the user file list yields a cluster with optimal resource load, further enhancing parsing efficiency. Calculating the real-time computing resources of each backend server and allocating the user file list to the target server cluster based on these resources creates a file allocation strategy. This strategy distributes multiple parsing tasks across multiple backend servers, preventing single-machine overload and improving parsing efficiency and accuracy. Parsing the allocated files in the user file list according to the allocation strategy, calculating the parsed file list and real-time computing resources, allows for iterative optimization of the allocation strategy. Real-time monitoring of backend server load and dynamic adjustment of the user file parsing strategy ensures balanced load across all backend servers. This rational utilization of the backend server cluster's computing resources effectively improves the stability and accuracy of user file parsing. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of an application environment for a user packet file parsing method based on load balancing, according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating a user packet file parsing method based on load balancing in one embodiment of the present invention; Figure 3 This is a schematic diagram of a user packet file parsing system based on load balancing in one embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] The user packet file parsing method based on load balancing provided in this invention can be applied to applications such as... Figure 1 In this application environment, the client communicates with the server via a network. The server can obtain user packet files from the client, split the user packet files to obtain a list of user files; select a target server cluster corresponding to the user file list from a preset backend server cluster based on the user file list; calculate the real-time computing resources of each backend server in the target server cluster, and allocate the user file list to the target server cluster according to the real-time computing resources, obtaining a file allocation strategy; parse the allocated files in the user file list according to the file allocation strategy, calculate the parsed file list and the real-time computing resources of the backend servers; iteratively optimize the file allocation strategy based on the real-time computing resources, until the iteratively optimized file allocation strategy can parse each user file in the parsed file list, obtaining the parsing result of the user packet file. User files can be allocated for file parsing reasonably according to the real-time load of the backend servers, thereby making reasonable use of the computing resources of the backend server cluster and effectively improving the stability and accuracy of user file parsing. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster composed of multiple servers. The present invention will now be described in detail through specific embodiments.

[0015] Please see Figure 2 As shown, Figure 2 A flowchart illustrating a load-balanced user packet file parsing method provided in an embodiment of the present invention includes the following steps: S1. Obtain the user package file, split the user package file to obtain a list of user files.

[0016] In this embodiment of the invention, a user package file is a collection of files created by a user for a specific software or system, used to extend functionality, customize settings, or store user data. For example, in the programming language Python, a user can create a package according to their needs, organize the relevant module files (.py) in a directory, and identify the directory as a package through the init.py file, thus obtaining the corresponding user package file.

[0017] Specifically, file splitting involves breaking down a large user package file into multiple smaller, more manageable, and usable files according to certain rules or requirements, based on the size of the user package file, resulting in a list of split user files.

[0018] In this embodiment of the invention, the step of splitting the user package file to obtain a user file list includes: Calculate the file size of the user package file, and calculate the number of file splits based on the file package size; The starting and ending byte positions for the file splitting are determined based on the number of file splits. The user file list is obtained by reading file data from the user package file based on the split start byte position and the split end byte position.

[0019] In this embodiment of the invention, the file size of each user packet file can be calculated based on the number of bytes occupied by the data type of each data item in the user packet file. For example, for a UTF-8 encoded user packet file, English characters and numbers occupy 1 byte, and Chinese characters occupy 3 bytes; for GBK encoding, Chinese characters occupy 2 bytes. The file size of the user packet file can be calculated by reading the number of characters in the text file and multiplying it by the number of bytes of the corresponding encoding. Then, the file size of the user packet file is compared with a preset file size threshold, and the ratio is rounded to the nearest integer, which is the number of file segments required to split each user packet file.

[0020] Furthermore, the user package files are divided according to the number of file splits to obtain the file split positions, namely the split start byte position and the split end byte position. Assuming the split file size is 1GB, or 1073741832 bytes, then the first split start byte position and split end byte position are bytes 0 to 1073741832. The second data block starts from position 1073741833. The end position of the second data block is calculated in the same way as the first data block. This process is repeated to split all user package files to obtain a list of user files.

[0021] Specifically, when splitting, the integrity of the data in the split files in the user file list should be considered. For example, assuming that the starting byte position and the ending byte position of the split are in the data, the starting byte position and the ending byte position of the split should be moved forward or backward to ensure the integrity of the data and avoid the fragmentation of a text or other type of data due to byte division.

[0022] In this embodiment of the invention, data is extracted from the user package file by splitting the start byte position and the end byte position to obtain multiple split files. The split files are then arranged and combined to obtain a user file list.

[0023] In this embodiment of the invention, by splitting the user package file, the split files in the user file list can be loaded one by one, avoiding excessive system resource consumption due to large files. At the same time, different split files can be parsed simultaneously, effectively improving the parsing effect of subsequent file packages.

[0024] S2. Select the target server cluster corresponding to the user file list from the preset backend server clusters according to the user file list; In this embodiment of the invention, the preset backend server cluster is a group of collaborative servers that can receive file parsing requests from clients or other systems and then parse the files accordingly.

[0025] Furthermore, however, each server in the backend server cluster may be faulty or running at full load, unable to allocate computing resources for parsing user packet files. Therefore, it is necessary to select servers from the backend server cluster that meet the computing load requirements to form the target server cluster. In this embodiment of the invention, selecting the target server cluster corresponding to the user file list from a preset backend server cluster based on the user file list includes: Obtain the running status of each backend server in the backend server cluster; Available backend servers are selected from the backend server cluster based on the operating status; Calculate the initial resource load for each of the available backend servers; The target server cluster corresponding to the user file list is selected from the available backend servers based on the initial resource load.

[0026] In this embodiment of the invention, the running status reflects whether the backend server can provide services normally, such as service running status, network usage status, security status, etc. The availability of backend servers can be determined by collecting the corresponding running status indicator data.

[0027] Furthermore, the initial resource load is the computational resource load of each backend server in the available backend servers before file parsing, that is, the computational resource usage status of each backend server. By using the initial computational load, we can determine the computational load that each backend server can handle in the future, thereby allocating resources more accurately.

[0028] In this embodiment of the invention, real-time load information of each available backend server can be collected, and the real-time load information can be weighted and calculated to obtain the initial resource load of each available backend server.

[0029] In this embodiment of the invention, when each available backend server receives a new file parsing task, each available backend server also carries historical tasks, resulting in a corresponding initial resource load. This load can be calculated by weighting real-time load information such as CPU utilization, memory utilization, bandwidth speed changes, and disk I / O utilization. The sum of the weights corresponding to the real-time load information is 1, and this can be preset.

[0030] In this embodiment of the invention, a target server cluster is formed by selecting available backend servers with initial resource loads less than a preset threshold. This results in a target server cluster with optimal resource load, thereby improving the parsing efficiency of subsequent user packet files.

[0031] S3. Calculate the real-time computing resources of each backend server in the target server cluster, and allocate the user file list to the target server cluster according to the real-time computing resources to obtain the file allocation strategy.

[0032] In this embodiment of the invention, real-time computing resources are the computing resources that each backend server in the target server cluster can use, including the utilization rate of the central processing unit, cache size, memory capacity, storage capacity, network resources, etc. The computing resources can be used to determine the ability of each backend server in the target server cluster to process data and run programs.

[0033] In this embodiment of the invention, information such as the CPU utilization rate, available cache size, available memory capacity, available storage capacity, and available network resources of each backend server in the target server cluster can be collected in real time to calculate computing resources. For example, the CPU utilization rate, available cache size, available memory capacity, available storage capacity, and available network resources can be weighted and calculated using preset weights to obtain real-time computing resources.

[0034] Furthermore, the file allocation strategy refers to how the target server cluster allocates file parsing tasks corresponding to the user file list, thereby performing file parsing on the user file list.

[0035] In this embodiment of the invention, the step of allocating the user file list to the target server cluster based on the real-time computing resources to obtain a file allocation strategy includes: Each user file in the user file list is parsed and categorized to obtain the file task category; The user files are prioritized according to the file task category to obtain the parsing priority; Based on the parsing priority and the real-time computing resources, the user files are allocated to obtain a file allocation strategy.

[0036] In this embodiment of the invention, the parsing task classification is based on the category of the parsing task corresponding to each user file. This can be determined according to the purpose and application scenario of the user package file when it is collected. For example, if the user file is used to store business data, such as sales data or user behavior data, the parsing task mainly extracts the data, cleans, transforms, and analyzes it to support decision-making and business optimization. If the user file is used to store application configuration parameters and settings, the parsing task reads the parameter values ​​in the configuration file to control the running behavior of the application.

[0037] Furthermore, priority division determines the order in which user files are parsed based on file task categories. Higher priority files are parsed first, and vice versa. The parsing priority can be obtained by dividing priority according to file task categories using preset priority rules.

[0038] In this embodiment of the invention, user files with high parsing priority can be preferentially allocated to target servers with larger real-time computing resources. User files are allocated sequentially according to parsing priority and computing resources until all user files in the user file list are allocated, thus obtaining a file allocation strategy.

[0039] In this embodiment of the invention, by constructing a file allocation strategy for the user file list, multiple user file parsing tasks corresponding to the user file list can be allocated to the corresponding backend servers. This can distribute multiple file parsing tasks to multiple backend servers, avoiding single-machine overload of the backend servers and effectively improving the efficiency and accuracy of file parsing.

[0040] S4. Based on the file allocation strategy, perform file parsing on the corresponding allocated files in the user file list, and calculate the parsed file list and the real-time parsing computing resources of the backend server.

[0041] In this embodiment of the invention, file parsing is performed according to the file task category of each user file, such as sales data, user behavior data, etc. The parsing task mainly extracts the data, cleans it, transforms it and analyzes it. When the user file is used to store the configuration parameters and settings of the application, the parsing task is to read the parameter values ​​in the configuration file and perform corresponding file parsing according to different file task categories to obtain the parsing result of the user package file.

[0042] Specifically, the step of parsing the allocated files in the user file list according to the file allocation strategy includes: The allocation file for each backend server in the target server cluster is determined from the user file list according to the file allocation policy. According to the file task category corresponding to the allocated file, the preset file parsing engine is invoked in the backend server; The allocated file is parsed using the file parsing engine.

[0043] In this embodiment of the invention, the user files allocated to each backend server can be determined through a file allocation strategy, i.e., the allocation files. Then, sequential file parsing is performed according to the parsing priority of the target user files. Each backend server has multiple file parsing engines preset to perform file parsing. The file parsing engines correspond to different file task categories and can perform file parsing on the user files corresponding to the file task categories to obtain the parsing results of the file allocation strategy.

[0044] Furthermore, a file parsing engine is a software component or module used to process and analyze data. It can read user files, parse their structure and content, and then process them according to predefined rules or algorithms to finally generate parsing results.

[0045] In detail, the list of parsed files after parsing the computational file and the real-time computing resources of the backend server include: Remove the allocated file from the user file list to obtain the parsed file list; The real-time computing resources for parsing each backend server are calculated based on the parsing results of the file parsing.

[0046] In this embodiment of the invention, the parsing results of the allocated files within a preset time period can be collected, and the real-time computing resources for parsing can be calculated based on the parsing results within the preset time period to iteratively optimize the file allocation strategy.

[0047] In this embodiment of the invention, parsing real-time computing resources refers to the real-time resource load of each backend server in the target server cluster after allocating user files and parsing them. This load load can be obtained by weighting and calculating the load information of each backend server, such as CPU utilization, memory utilization, bandwidth speed change, and disk I / O utilization, after real-time collection of the parsing results.

[0048] Specifically, real-time load information such as CPU usage, memory usage, bandwidth speed changes, and disk I / O usage of the backend server can be collected in real time through Task Manager, performance monitor, and preset data collection scripts. Then, the real-time computing resources of the backend server can be calculated and analyzed using the methods described above.

[0049] Preferably, the file parsing speed of user files in the user file list can be monitored in real time. If the file parsing is too slow, the parsing speed of user files can be further accelerated by dynamically adding backend servers.

[0050] In this embodiment of the invention, by calculating and parsing real-time computing resources, the load of the backend server can be monitored in real time, and the file parsing strategy of user files can be dynamically adjusted to ensure the load balance of each backend server. At the same time, when a backend server fails, the tasks of the backend server can be automatically redistributed to other normal backend servers to ensure the continuity and stability of the parsing tasks.

[0051] S5. Iteratively optimize the file allocation strategy based on the real-time computing resources for parsing, until the iteratively optimized file allocation strategy parses each user file in the parsing file list to obtain the parsing result of the user package file.

[0052] In this embodiment of the invention, iterative optimization involves real-time monitoring of the load on the backend servers and dynamically adjusting the file allocation strategy for user files based on the real-time load, thereby ensuring load balance among the backend servers in the target server cluster.

[0053] Specifically, the step of iteratively optimizing the file allocation strategy based on the real-time computing resources for parsing, until the iteratively optimized file allocation strategy parses each user file in the parsing file list to obtain the parsing result of the user package file, includes: Based on the analysis of real-time computing resources, an updated file allocation strategy is generated; The updated file allocation strategy is used as the file allocation strategy, and the steps described above for parsing the corresponding allocated files in the user file list according to the file allocation strategy are returned. Each user file in the parsing file list is parsed until all user files in the parsing file list have been parsed, and the parsing result of the user package file is obtained.

[0054] In detail, based on the real-time computing resources for parsing, server allocation is performed on the unparsed user files in the parsed file list to obtain an updated file allocation strategy.

[0055] The updated file allocation strategy can be calculated based on the above method for allocating user files. Specifically, it determines the parsing priority of unparsed files in the parsing file list, and allocates unparsed files sequentially according to the parsing priority and computing resources to obtain the updated file allocation strategy.

[0056] Furthermore, the updated file allocation strategy is used as the file allocation strategy, and the steps described above for parsing the allocated files in the user file list according to the file allocation strategy are returned. This process iterates through the parsing of user files in the file list using the updated file allocation strategy, repeating the process until all user files in the file list have been parsed, resulting in the parsing result for each user file in the user package file. Specifically, a preset file parsing engine is called in the backend server to parse the user files according to the updated and iterated file allocation strategy, obtaining the parsing result for the user package file.

[0057] In this embodiment of the invention, by updating the file allocation strategy to parse the file list, user files can be reasonably allocated for parsing based on the real-time load of the backend server, thereby making reasonable use of the computing resources of the backend server cluster and effectively improving the stability and accuracy of user file parsing.

[0058] As can be seen, in the above scheme, by splitting the user package file, the split files in the user file list can be loaded one by one, and different split files can be parsed simultaneously, effectively improving the parsing effect of subsequent file packages. Selecting the target server cluster based on the user file list can obtain the target server cluster with optimal resource load, thereby improving the parsing efficiency of subsequent user package files. Calculating the real-time computing resources of each backend server and allocating the user file list to the target server cluster according to the real-time computing resources yields a file allocation strategy. This strategy can distribute multiple file parsing tasks across multiple backend servers, avoiding single-machine overload and effectively improving the efficiency and accuracy of file parsing. Parsing the corresponding allocated files in the user file list according to the file allocation strategy, calculating the parsed file list and real-time computing resources, allows for iterative optimization of the file allocation strategy. Real-time monitoring of the backend server load and dynamic adjustment of the user file parsing strategy ensures load balance across all backend servers. Thus, the computing resources of the backend server cluster are utilized rationally, effectively improving the stability and accuracy of user file parsing.

[0059] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0060] In one embodiment, a load-balanced user packet file parsing system is provided, which corresponds one-to-one with the load-balanced user packet file parsing method described in the above embodiments. For example... Figure 3 As shown, the load-balanced user packet file parsing system includes a file splitting module 101, a server cluster selection module 102, a file allocation strategy construction module 103, a file parsing module 104, and an iterative optimization module 105. Detailed descriptions of each functional module are as follows: The file splitting module 101 is used to obtain user package files, split the user package files, and obtain a list of user files. The server cluster selection module 102 is used to select the target server cluster corresponding to the user file list from the preset backend server clusters according to the user file list. The file allocation strategy construction module 103 is used to calculate the real-time computing resources of each backend server in the target server cluster, and allocate the user file list to the target server cluster according to the real-time computing resources to obtain the file allocation strategy. The file parsing module 104 is used to parse the allocated files in the user file list according to the file allocation strategy, and to calculate the parsed file list and the real-time parsing computing resources of the backend server. The iterative optimization module 105 is used to iteratively optimize the file allocation strategy based on the real-time computing resources for parsing, until the iteratively optimized file allocation strategy parses each user file in the parsing file list to obtain the parsing result of the user package file.

[0061] In one embodiment, the file splitting module 101 splits the user package file to obtain a user file list, including: Calculate the file size of the user package file, and calculate the number of file splits based on the file package size; The starting and ending byte positions for the file splitting are determined based on the number of file splits. The user file list is obtained by reading file data from the user package file based on the split start byte position and the split end byte position.

[0062] In one embodiment, the server cluster selection module 102 selects the target server cluster corresponding to the user file list from a preset backend server cluster based on the user file list, including: Obtain the running status of each backend server in the backend server cluster; Available backend servers are selected from the backend server cluster based on the operating status; Calculate the initial resource load for each of the available backend servers; The target server cluster corresponding to the user file list is selected from the available backend servers based on the initial resource load.

[0063] In one embodiment, the file allocation strategy construction module 103 obtains a file allocation strategy by allocating the user file list to the target server cluster based on the computing resources, including: Each user file in the user file list is parsed and categorized to obtain the file task category; The user files are prioritized according to the file task category to obtain the parsing priority; Based on the parsing priority and the computing resources, the user files are allocated to obtain a file allocation strategy.

[0064] In one embodiment, the file parsing module 104 parses the allocated files in the user file list according to the file allocation strategy, including: The allocation file for each backend server in the target server cluster is determined from the user file list according to the file allocation policy. According to the file task category corresponding to the allocated file, the preset file parsing engine is invoked in the backend server; The allocated file is parsed using the file parsing engine.

[0065] In one embodiment, the file parsing module 104 calculates the parsed file list and the real-time parsing computing resources of the backend server, including: Remove the allocated file from the user file list to obtain the parsed file list; The real-time computing resources for parsing each backend server are calculated based on the parsing results of the parsed file.

[0066] In one embodiment, the iterative optimization module 105 iteratively optimizes the file allocation strategy based on the real-time computing resources for parsing, until the iteratively optimized file allocation strategy parses each user file in the parsing file list to obtain the parsing result of the user package file, including: Based on the analysis of real-time computing resources, an updated file allocation strategy is generated; The updated file allocation strategy is used as the file allocation strategy, and the steps described above for parsing the corresponding allocated files in the user file list according to the file allocation strategy are returned. Each user file in the parsing file list is parsed until all user files in the parsing file list have been parsed, and the parsing result of the user package file is obtained.

[0067] This invention provides a load-balanced user packet file parsing system. By splitting user packet files, the system can load the split files from the user file list one by one and parse different split files simultaneously, effectively improving the parsing efficiency of subsequent file packets. Selecting a target server cluster based on the user file list yields a cluster with optimal resource load, further improving the parsing efficiency of subsequent user packet files. Calculating the real-time computing resources of each backend server and constructing a file allocation strategy based on these resources allows multiple file parsing tasks to be distributed across multiple backend servers, preventing single-machine overload and effectively improving parsing efficiency and accuracy. Parsing the allocated files in the user file list according to the allocation strategy, calculating the parsed file list and real-time computing resources, allows for iterative optimization of the file allocation strategy. Real-time monitoring of backend server load and dynamic adjustment of the user file parsing strategy ensures load balance across all backend servers. This rationally utilizes the computing resources of the backend server cluster, effectively improving the stability and accuracy of user file parsing.

[0068] For specific limitations regarding the load-balanced user packet file parsing system, please refer to the limitations of the load-balanced user packet file parsing method described above, which will not be repeated here. Each module in the aforementioned load-balanced user packet file parsing system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0069] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a load-balanced user packet file parsing method on the server side.

[0070] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input system connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a load-balanced user packet file parsing method. In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the user package file, split the user package file to obtain a list of user files; Based on the user file list, select the target server cluster corresponding to the user file list from the preset backend server cluster; Calculate the real-time computing resources of each backend server in the target server cluster, and allocate the user file list to the target server cluster according to the real-time computing resources to obtain a file allocation strategy; According to the file allocation strategy, the allocated files in the user file list are parsed, and the parsed file list and the real-time parsing computing resources of the backend server are calculated. The file allocation strategy is iteratively optimized based on the real-time computing resources for parsing, until the optimized file allocation strategy parses each user file in the parsing file list to obtain the parsing result of the user package file.

[0071] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain the user package file, split the user package file to obtain a list of user files; Based on the user file list, select the target server cluster corresponding to the user file list from the preset backend server cluster; Calculate the real-time computing resources of each backend server in the target server cluster, and allocate the user file list to the target server cluster according to the real-time computing resources to obtain a file allocation strategy; According to the file allocation strategy, the allocated files in the user file list are parsed, and the parsed file list and the real-time parsing computing resources of the backend server are calculated. The file allocation strategy is iteratively optimized based on the real-time computing resources for parsing, until the optimized file allocation strategy parses each user file in the parsing file list to obtain the parsing result of the user package file.

[0072] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0073] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0074] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.

[0075] Finally, it should be noted that if any software tools or components not belonging to this company appear in the embodiments of the application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A user packet file parsing method based on load balancing, characterized in that, include: Obtain the user package file, split the user package file to obtain a list of user files; Select the target server cluster corresponding to the user file list from the preset backend server cluster; Calculate the real-time computing resources of each backend server in the target server cluster, and allocate the user file list to the target server cluster according to the real-time computing resources to obtain a file allocation strategy; According to the file allocation strategy, the allocated files in the user file list are parsed, and the parsed file list and the real-time parsing computing resources of the backend server are calculated. The file allocation strategy is iteratively optimized based on the real-time computing resources for parsing, until the optimized file allocation strategy parses each user file in the parsing file list to obtain the parsing result of the user package file. The file allocation strategy is dynamically optimized based on the real-time computing resources of the target server cluster to obtain an optimized file allocation strategy. The optimized file allocation strategy is used to parse each user file in the user file list to obtain the parsing result of the user package file.

2. The user packet file parsing method based on load balancing as described in claim 1, characterized in that, The process of splitting the user package file to obtain a list of user files includes: Calculate the file size of the user package file, and calculate the number of file splits based on the file package size; The starting and ending byte positions for the file splitting are determined based on the number of file splits. The user file list is obtained by reading file data from the user package file based on the split start byte position and the split end byte position.

3. The user packet file parsing method based on load balancing as described in claim 1, characterized in that, Selecting the target server cluster corresponding to the user file list from the preset backend server cluster includes: Obtain the running status of each backend server in the backend server cluster; Available backend servers are selected from the backend server cluster based on the operating status; Calculate the initial resource load for each of the available backend servers; The target server cluster corresponding to the user file list is selected from the available backend servers based on the initial resource load.

4. The user packet file parsing method based on load balancing as described in claim 1, characterized in that, The step of allocating the user file list to the target server cluster based on the computing resources to obtain a file allocation strategy includes: Each user file in the user file list is parsed and categorized to obtain the file task category; The user files are prioritized according to the file task category to obtain the parsing priority; Based on the parsing priority and the computing resources, the user files are allocated to obtain a file allocation strategy.

5. The user packet file parsing method based on load balancing as described in claim 1, characterized in that, The step of parsing the allocated files in the user file list according to the file allocation strategy includes: The allocation file for each backend server in the target server cluster is determined from the user file list according to the file allocation policy. According to the file task category corresponding to the allocated file, the preset file parsing engine is invoked in the backend server; The allocated file is parsed using the file parsing engine.

6. The user packet file parsing method based on load balancing as described in claim 1, characterized in that, The list of parsed files after parsing the computational file and the real-time computational resources of the backend server include: Remove the allocated file from the user file list to obtain the parsed file list; The real-time computing resources for parsing each backend server are calculated based on the parsing results of the parsed file.

7. The user packet file parsing method based on load balancing as described in claim 1, characterized in that, The step of iteratively optimizing the file allocation strategy based on the real-time computing resources for parsing, until the iteratively optimized file allocation strategy parses each user file in the parsing file list to obtain the parsing result of the user package file, includes: Based on the analysis of real-time computing resources, an updated file allocation strategy is generated; The updated file allocation strategy is used as the file allocation strategy, and the steps described above for parsing the corresponding allocated files in the user file list according to the file allocation strategy are returned. Each user file in the parsing file list is parsed until all user files in the parsing file list have been parsed, and the parsing result of the user package file is obtained.

8. A user packet file parsing system based on load balancing, characterized in that, include: The file splitting module is used to obtain user package files, split the user package files, and obtain a list of user files. The server cluster selection module is used to select the target server cluster corresponding to the user file list from the preset backend server clusters based on the user file list. The file allocation strategy construction module is used to calculate the real-time computing resources of each backend server in the target server cluster, and allocate the user file list to the target server cluster according to the real-time computing resources to obtain the file allocation strategy. The file parsing module is used to parse the allocated files in the user file list according to the file allocation strategy, and to calculate the parsed file list and the real-time parsing computing resources of the backend server. The iterative optimization module is used to iteratively optimize the file allocation strategy based on the real-time computing resources for parsing, until the iteratively optimized file allocation strategy parses each user file in the parsing file list to obtain the parsing result of the user package file.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the load-balanced user packet file parsing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the load-balanced user packet file parsing method as described in any one of claims 1 to 7.

Images