Object list data splitting method and device, electronic equipment and storage medium

Abstract

The invention provides an object list data splitting method and device, electronic equipment and a storage medium, and the object list data splitting method comprises the steps: obtaining to-be-split target object list data, and extracting data feature information, business rule information and resource availability information of the target object list data; determining a target splitting strategy of the target object list data based on the data feature information, the business rule information and the resource availability information of the target object list data; and splitting the target object list data according to the target splitting strategy to obtain a target splitting result of the target object list data. Thus, by analyzing the data feature information, the business rule information and the resource availability information of the object list data, a proper splitting strategy can be matched for the object list data, and the splitting efficiency of the object list data is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, electronic device, and storage medium for splitting object list data. Background Art

[0002] Object list data refers to a collection of multiple objects, each of which has specific properties and structures, such as List in Java. <bean>In order to process this data efficiently, it is often necessary to split it into multiple sublists.

[0003] Existing methods for splitting object list data typically require developers to manually write loops and conditional logic to process each object one by one and assign it to different sublists. When faced with large-scale data or complex structures, this manual splitting method is lengthy and error-prone, resulting in low object list data splitting efficiency. Summary of the Invention

[0004] In view of this, the embodiments of the present application at least provide a method, device, electronic device and storage medium for splitting object list data. By analyzing the data feature information, business rule information and resource availability information of the object list data, a suitable splitting strategy can be matched for the object list data, thereby improving the splitting efficiency of the object list data.

[0005] This application mainly includes the following aspects: In a first aspect, an embodiment of the present application provides a method for splitting object list data, the method comprising: Acquire target object list data to be split, and extract data feature information, business rule information, and resource availability information of the target object list data; Determining a target splitting strategy for the target object list data based on data feature information, business rule information, and resource availability information of the target object list data; The target object list data is split according to the target splitting strategy to obtain a target splitting result of the target object list data.

[0006] In a second aspect, an embodiment of the present application further provides a device for splitting object list data, the device for splitting object list data comprising: A data acquisition module is used to acquire target object list data to be split and extract data feature information, business rule information and resource availability information of the target object list data; a strategy determination module, configured to determine a target splitting strategy for the target object list data based on data feature information, business rule information, and resource availability information of the target object list data; The data splitting module is used to split the target object list data according to the target splitting strategy to obtain a target splitting result of the target object list data.

[0007] In a third aspect, an embodiment of the present application further provides an electronic device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the object list data splitting method described above.

[0008] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the object list data splitting method described above are executed.

[0009] The present application provides a method, device, electronic device, and storage medium for splitting object list data. The object list data splitting method includes: obtaining target object list data to be split, and extracting data feature information, business rule information, and resource availability information of the target object list data; determining a target splitting strategy for the target object list data based on the data feature information, business rule information, and resource availability information of the target object list data; and splitting the target object list data according to the target splitting strategy to obtain a target splitting result for the target object list data. Thus, by analyzing the data feature information, business rule information, and resource availability information of the object list data, a suitable splitting strategy can be matched to the object list data, thereby improving the efficiency of splitting the object list data.

[0010] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A flowchart of a method for splitting object list data provided in an embodiment of the present application is shown; Figure 2 One of the functional module diagrams of a device for splitting object list data provided in an embodiment of the present application is shown; Figure 3 A second functional module diagram of a device for splitting object list data provided in an embodiment of the present application is shown; Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0014] To facilitate understanding of the present application, the technical solutions provided in the present application are described in detail below in conjunction with specific embodiments.

[0015] See also Figure 1 , Figure 1 This is a flow chart of a method for splitting object list data provided in an embodiment of the present application. Figure 1 As shown, the object list data splitting method provided in the embodiment of the present application includes the following steps: S101 , obtaining target object list data to be split, and extracting data feature information, business rule information, and resource availability information of the target object list data.

[0016] Here, first, we need to obtain the target object list data to be split. The target object list data refers to a collection of multiple objects, each of which has specific properties and structures. These data usually come from business systems, databases or data files for subsequent processing and analysis. In this embodiment of the application, the target object list data is a List<T> in Java. <bean>Data. In order to process these data efficiently, it is necessary to extract three types of information related to splitting from the target object list data, namely data feature information, business rule information and resource availability information. Among them, the data feature information includes the list length and list structure type of the target object list data, which is used to describe the scale and composition structure of the data. The business rule information includes at least one business rule, that is, the splitting condition of the target object list data, which is used to guide the specific method of data splitting. The resource availability information includes the current number of available CPU cores, the current CPU load and the remaining heap memory of the target device used to split the target object list data, which is used to evaluate whether the system resources support operations such as parallel processing. By extracting this information, a comprehensive support basis can be provided for subsequent splitting strategy decisions.

[0017] S102: Determine a target splitting strategy for the target object list data based on data feature information, business rule information, and resource availability information of the target object list data.

[0018] Here, after acquiring the target object list data and related information, a target splitting strategy is determined based on the target object list data's data characteristics, business rules, and resource availability. This process comprehensively considers data characteristics, business requirements, and system resources to select the most appropriate splitting method. By analyzing the data's size, structure, business rules, and system resource availability, the most appropriate splitting strategy for the target object list data can be dynamically determined, ensuring that the splitting process is both efficient and meets business requirements, thereby improving splitting efficiency and ensuring splitting accuracy.

[0019] S103: Split the target object list data according to the target splitting strategy to obtain a target splitting result of the target object list data.

[0020] Here, based on the determined target splitting strategy, a specific splitting operation is performed on the target object list data. This process divides the target object list data into multiple sublists, each containing a collection of objects that meet specific criteria. This method can break down large or complex data into smaller, more manageable chunks, improving data processing efficiency and flexibility. The final result is the target splitting result for the target object list data.

[0021] Furthermore, the resource availability information of the target object list data is extracted according to the following steps: Step a1, obtaining the current number of available CPU cores, current CPU load, and remaining heap memory of the target device; the target device is the device that executes the object list data splitting method.

[0022] Here, the target device refers to the computer device that actually runs the object list data splitting method. Use the "Runtime.getRuntime().availableProcessors()" method to obtain the current number of available CPU cores of the device through the system interface or monitoring tools. , Current CPU load (usually a value between 0 and 1, indicating the current CPU usage) and the remaining heap memory (Unit: MB). This information reflects the resource status of the target device when the split operation is performed.

[0023] Step a2: Calculate the parallel feasibility score of the target object list data according to the current number of available CPU cores, current CPU load, and remaining heap memory of the target device.

[0024] Here, the parallel feasibility score Is a comprehensive indicator used to evaluate whether the target device is suitable for parallel processing. The score can be calculated using the following formula: .

[0025] This formula takes into account the CPU load , number of available cores And the remaining heap memory , to determine the parallel processing capabilities of the target device in its current state.

[0026] Step a3: Determine the remaining heap memory of the target device and the parallel feasibility score of the target object list data as resource availability information of the target object list data.

[0027] Here, the resource availability information of the target object list data includes two key indicators: the remaining heap memory of the target device and parallel feasibility score These two metrics together reflect the resource status and parallel processing capabilities of the target device when performing the object list data splitting operation. By determining these two metrics as resource availability information, they can provide important reference for subsequent splitting strategy decisions.

[0028] Furthermore, determining the target splitting strategy of the target object list data based on the data feature information, business rule information and resource availability information of the target object list data includes: Step b1: judging whether the target object list data has a nested structure based on data characteristic information of the target object list data; the data characteristic information includes the list length and list structure type of the target object list data.

[0029] Here, the list length in the data feature information The list structure type is used to evaluate the complexity of the target object list data. List length The list structure type describes the data structure, especially whether there are nested sublists or mapping structures. By analyzing this information, we can determine whether the data has a nested structure, thus providing a basis for the subsequent target splitting strategy selection.

[0030] Step b2: If yes, determining a target splitting strategy for the target object list data based on the parallel feasibility score of the target object list data, the list length, and the remaining heap memory of the target device.

[0031] Here, if the target object list data has a nested structure, the parallel feasibility score needs to be further considered. , list length and the remaining heap memory of the target device Parallel feasibility score Reflects the parallel processing capability of the target device in its current state. The length of the list Indicates the size of the data and the remaining heap memory This reflects the memory resource status of the system. Based on this information, the most suitable splitting strategy can be determined, such as whether to use parallel splitting or recursive splitting.

[0032] Step b3: If not, determining a target splitting strategy for the target object list data based on the parallel feasibility score, list length, and business rule information of the target object list data.

[0033] Here, if the target object list data does not have a nested structure, the parallel feasibility score of the target object list data is mainly considered. , list length and business rule information. Business rule information includes at least one business rule that guides how to split data based on specific conditions. By comprehensively considering these factors, the most appropriate splitting strategy can be determined, such as whether to use parallel splitting or dynamic splitting.

[0034] Furthermore, determining a target splitting strategy for the target object list data based on the parallel feasibility score of the target object list data, the list length, and the remaining heap memory of the target device includes: Step c1: If the parallel feasibility score is greater than a preset feasibility score threshold, and the list length is greater than a preset first list length threshold, determining that the target splitting strategy includes a parallel splitting strategy and a recursive splitting strategy.

[0035] Here, when the parallel feasibility score Greater than the preset feasibility score threshold , and the length of the target object list data Greater than the preset first list length threshold , indicating that the target device has sufficient computing resources to process large-scale data, and the data volume is large enough to benefit from parallel processing. Therefore, the target splitting strategy is determined to be a combination of parallel splitting strategy and recursive splitting strategy. This combination strategy can make full use of the computing power of multi-core processors, process nested structures at the same time, and improve splitting efficiency. In this embodiment of the application, the feasibility score threshold The value is 0.7, the first list length threshold The value is 10000.

[0036] Step c2: If the parallel feasibility score is less than or equal to the feasibility score threshold or the list length is less than or equal to the first list length threshold, and the remaining heap memory of the target device is greater than the preset remaining heap memory threshold, then determine that the target splitting strategy is a recursive splitting strategy.

[0037] Here, when the parallel feasibility score Less than or equal to the feasibility score threshold , or the length of the list Less than or equal to the first list length threshold , but the remaining heap memory of the target device Greater than the preset remaining heap memory threshold , indicating that although the conditions for parallel processing are not met, the device memory is sufficient to process the nested structure. Therefore, the target splitting strategy is determined to be a recursive splitting strategy. The recursive splitting strategy is suitable for processing nested structures and can decompose data layer by layer to ensure efficient processing. In this embodiment of the application, the remaining heap memory threshold The value is 512MB.

[0038] Step c3: If the parallel feasibility score is less than or equal to the feasibility score threshold or the list length is less than or equal to the first list length threshold, and the remaining heap memory of the target device is less than or equal to the remaining heap memory threshold, then determine that the target splitting strategy is an iterative simulation recursive splitting strategy.

[0039] Here, when the parallel feasibility score Less than or equal to the feasibility score threshold , and the list length Less than or equal to the first list length threshold , while the remaining heap memory of the target device Less than or equal to the remaining heap memory threshold , indicating that the target device has limited computing and memory resources. In this case, the target splitting strategy is determined to be the iterative simulation recursive splitting strategy. This strategy simulates the recursive process iteratively, avoiding stack overflows that can be caused by recursive calls while conserving memory and is suitable for resource-constrained environments.

[0040] Furthermore, determining a target splitting strategy for the target object list data based on the parallel feasibility score, list length, and business rule information of the target object list data includes: Step d1: If the list length is greater than a preset second list length threshold, and the parallel feasibility score is greater than the feasibility score threshold, determining that the target splitting strategy is a parallel splitting strategy.

[0041] Here, when the length of the target object list data Greater than the preset second list length threshold , and the parallel feasibility score Greater than the preset feasibility score threshold , indicating that the data volume is large and the target device has sufficient computing resources to support parallel processing. Therefore, the target splitting strategy is determined to be a parallel splitting strategy. The parallel splitting strategy can make full use of the computing power of the multi-core processor and significantly improve the splitting efficiency. In this embodiment of the application, the second list length threshold The value is 50000.

[0042] Step d2: If the list length is greater than the second list length threshold and the parallel feasibility score is less than or equal to the feasibility score threshold and the number of business rules in the business rule information is greater than or equal to the preset business rule number threshold, or the list length is less than or equal to the second list length threshold and the number of business rules is greater than or equal to the business rule number threshold, then the target splitting strategy is determined to be a dynamic splitting strategy.

[0043] Here, when the list length Greater than the second list length threshold But the parallel feasibility score The parallel processing conditions are not met (i.e. ), and the number of business rules in the business rule information Greater than or equal to the preset business rule quantity threshold , or when the list length Less than or equal to the second list length threshold But the number of business rules Still greater than or equal to the threshold number of business rules , indicating that although the parallel processing conditions are not met, the business rules are complex and need to be handled flexibly. Therefore, the target splitting strategy is determined to be a dynamic splitting strategy. The dynamic splitting strategy can dynamically adjust the splitting logic according to the business rules and is suitable for complex business scenarios. Among them, the business rule number threshold The value of is 2.

[0044] In the embodiment of the present application, the core goal of the dynamic splitting strategy is to process data flexibly and efficiently, aiming to sort the List according to multiple conditions defined by the developer. <bean>The core design is based on the interface, allowing developers to implement "SplitCondition <t>"The interface customizes the splitting conditions corresponding to the business rules (such as by the name attribute). By traversing all splitting conditions, matching each element, and adding the elements that meet the conditions to the corresponding sublist, a List is finally generated. <List <bean>>The target splitting result of the structure. The specific steps are as follows: First, traverse the splitting conditions of all business rules in the target object list data (each splitting condition is an independent logical unit for judging whether an element meets a specific condition. For example, condition 1: "name=Zhang San", condition 2: "age=18"). Then, for each element in the target object list data, check whether the element meets the above splitting conditions. If it meets the conditions (the elements that meet one condition are put into one sub-list, and the others are put into another list), add the element to the corresponding sub-list, and finally obtain the target splitting result of the target object list data.

[0045] Step d3, if the length of the list is greater than the second list length threshold and the parallel feasibility score is less than or equal to the feasibility score threshold and the number of business rules is less than the business rule number threshold, or the length of the list is less than or equal to the second list length threshold and the number of business rules is less than the business rule number threshold, then determine that the target splitting strategy is the basic splitting strategy.

[0046] Here, when the length of the list is greater than the second list length threshold but the parallel feasibility score does not meet the parallel processing condition (that is ), and the number of business rules is less than the business rule number threshold , or when the length of the list is less than or equal to the second list length threshold and the number of business rules is also less than the business rule number threshold , it means that the data volume of the target object list data is small and the business rules are simple, and no complex splitting logic is required. Therefore, determine that the target splitting strategy is the basic splitting strategy.

[0047] In the embodiments of the present application, the basic splitting strategy can adopt any one of the following splitting strategies: Manual loop splitting, using a custom-written tool class to implement the splitting function of the List, and combining loops and conditional judgments. Manually write a loop to traverse the List and split the elements into multiple sub-lists according to specific conditions or a fixed size.

[0048] Adopt the "ListUtils.partition()" method provided by Apache Commons Collections or the "Lists.partition()" method provided by Guava to split the List into multiple sub-lists according to a fixed size.

[0049] Use the Stream API introduced in Java 8 and the "Collectors.groupingBy()" or "Collectors.partitioningBy()" method to conditionally split the List.

[0050] Furthermore, when the target splitting strategy includes a parallel splitting strategy, splitting the target object list data according to the target splitting strategy to obtain a target splitting result of the target object list data includes: Step e1: determining the number of split tasks of the parallel splitting strategy according to the number of available CPU cores of the target device and the list length of the target object list data.

[0051] Here, the core goal of the parallel splitting strategy is to split large-scale data into multiple sub-lists through task sharding, multi-threaded parallel processing and result merging, and finally merge the results after all thread tasks are completed. The determination is based on the number of available CPU cores of the target device and the list length of the target object list data By dynamically adjusting the number of split tasks, the efficiency of parallel processing and the rationality of resource utilization can be ensured.

[0052] Specifically, the number of split tasks is determined according to the following rules: : like When the target object list data is determined to be extremely small, in order to avoid wasting resources, determine .

[0053] like , then determine that the amount of data in the target object list is moderate. and Balanced, the default sharding setting , that is, 1 core and 1 thread.

[0054] like , it is determined that the target object list data volume is large, and the number of split tasks needs to be dynamically expanded. At this time, set ,in, is the expansion coefficient, Round up, the upper limit is Constraints. For example, when =4, =40000, calculate the expansion factor = 2 (because 40000 / (5000×4)=2), so we can determine T =8.

[0055] In this way, by dynamically adjusting the number of split tasks, we can avoid over-sharding and blockage caused by allocating too much data to a single thread, ensuring the efficiency of parallel processing and the rationality of resource utilization.

[0056] Specifically, the input data List <t>Divide into multiple subtasks, each subtask is processed by a thread. The sharding strategy can dynamically adjust the number of shards according to the size of the thread pool. For example, if the number of threads in the thread pool is 4, the input data will be divided into 4 sublists.

[0057] Step e2: Create a thread pool with the same number of threads as the number of split tasks, and divide the target object list data into multiple subtasks according to the number of split tasks.

[0058] Here, we first use ExecutorService to create a fixed-size thread pool with a number of threads equal to the number of split tasks. This can be expressed as "ExecutorService executor = Executors.newFixedThreadPool(threadPoolSize)"; "threadPoolSize" is the number of threads in the thread pool. The use of a thread pool effectively manages thread resources and improves the efficiency of parallel processing. Next, we divide the target object list data into multiple subtasks according to the number of split tasks, ensuring that each subtask contains roughly the same amount of data.

[0059] Step e3: Split the multiple subtasks in parallel according to the threads in the thread pool to obtain a split result for each subtask.

[0060] Here, threads in the thread pool are used to process each subtask in parallel. Each thread independently performs the subtask splitting operation, thereby fully utilizing the computing power of the multi-core processor. Specifically, you can use "Future <List<List <t>>>future = executor.submit(() ->splitSubList(subList, conditions))" to manage the thread pool and submit subtasks for parallel processing.

[0061] Step e4: collecting the splitting results of each of the subtasks and merging them to obtain the target splitting result of the target object list data.

[0062] Here, the split results of all subtasks are collected and merged to obtain the final target split result. This can be achieved by waiting for all subtasks to complete and collecting their results. Finally, the split results of all subtasks are merged into a list of lists, where each sublist contains elements that meet a specific condition. In this embodiment of the application, a "Future" object is used to collect the split results of each thread "List <List <t>>subResult = future.get();result.addAll(subResult);", wait for all thread tasks to be executed and shut down the thread pool ("executor.shutdown()"); and merge all child threads into the corresponding sublist.

[0063] Furthermore, when the target splitting strategy includes a recursive splitting strategy, splitting the target object list data according to the target splitting strategy to obtain a target splitting result of the target object list data includes: Step f1, traversing each element of the target object list data, and determining the list structure type of each element; the list structure type includes a nested structure and a non-nested structure.

[0064] The core goal of the recursive splitting strategy is to deeply traverse nested data structures and split common objects (such as beans) based on splitting criteria (for example, splitting by the attribute value "name" so that all "name" elements consistently enter the same sublist). For Lists and Maps (where element type identification is determined using the "instanceof" keyword, for example, to determine the List type: "instanceof List" for the current element), the recursive splitting strategy continues to traverse their child elements rather than directly splitting them. Based on this, the recursive splitting strategy traverses the input data and splits based on the criteria. Specifically, it first traverses each element in the target object's list data to determine its list structure type. List structures can be nested or non-nested. List structures can be nested lists (e.g., a list of objects containing another list of objects), maps (e.g., a map of objects containing another list of objects or a map of objects), or non-nested structures, i.e., common objects such as beans. By identifying these list structure types, further processing of each element can be determined.

[0065] Step f2: for any element of the target object list data, if the list structure type of the element is a list structure in a nested structure, traverse each child element in the list structure of the element and recursively call the recursive splitting strategy for each child element.

[0066] Here, if the current element is a list structure, it is necessary to further traverse each child element in the list. For each child element, the recursive splitting strategy is recursively called to handle potentially deeper nested structures. In the recursion, the child element Bean is also split based on the splitting condition (such as the attribute value "name"), ensuring comprehensive processing of multi-level nested structures.

[0067] Step f3: If the list structure type of the element is a mapping structure in a nested structure, traverse each value in the mapping structure of the element and recursively call the recursive splitting strategy for each value.

[0068] If the current element is a Map, each value in the resulting Map must be traversed. For each value, the recursive splitting strategy is recursively invoked to handle possible nested structures. Child beans are also split based on splitting criteria (such as the attribute value "name") during the recursion, ensuring deep processing of nested data in the Map.

[0069] Step f4: if the list structure type of the element is a non-nested structure, the elements are matched according to the business rules in the target object list data business rule information, and the elements that meet the business rules are added to the sublist corresponding to the business rules.

[0070] Here, if the current element is a plain object (such as a bean) without a nested structure, the element is matched based on the business rule splitting conditions (such as the attribute value "name") in the business rule information. If the element meets the splitting conditions of a business rule, it is added to the corresponding sublist.

[0071] Step f5: merging the sublists obtained by the recursive splitting to obtain a target splitting result of the target object list data.

[0072] Here, all sublists obtained during the recursive splitting process are merged to finally obtain the target split result of the target object list data. This step ensures that all processed data are integrated together to form a List <List <bean>The final target split result of the structure, where each sublist contains elements that meet the specific splitting conditions.

[0073] An embodiment of the present application provides a method for splitting object list data, comprising: obtaining target object list data to be split, and extracting data feature information, business rule information, and resource availability information of the target object list data; determining a target splitting strategy for the target object list data based on the data feature information, business rule information, and resource availability information of the target object list data; and splitting the target object list data according to the target splitting strategy to obtain a target splitting result for the target object list data. Thus, by analyzing the data feature information, business rule information, and resource availability information of the object list data, a suitable splitting strategy can be matched to the object list data, thereby improving the efficiency of object list data splitting.

[0074] Based on the same application concept, the embodiments of the present application also provide an object list data splitting device corresponding to the object list data splitting method provided in the above embodiments. Since the principle of solving the problem by the device in the embodiments of the present application is similar to the object list data splitting method in the above embodiments of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0075] See also Figure 2 , Figure 2 This is one of the functional module diagrams of a device for splitting object list data provided in an embodiment of the present application. Figure 2 As shown, the object list data splitting device 200 includes: The data acquisition module 210 is used to acquire target object list data to be split, and extract data feature information, business rule information and resource availability information of the target object list data.

[0076] The strategy determination module 220 is configured to determine a target splitting strategy for the target object list data based on data feature information, business rule information, and resource availability information of the target object list data.

[0077] The data splitting module 230 is configured to split the target object list data according to the target splitting strategy to obtain a target splitting result of the target object list data.

[0078] Further, see Figure 3 , Figure 3 This is the second functional module diagram of a device for splitting object list data provided in an embodiment of the present application. Figure 3 As shown, the object list data splitting device 200 further includes: The device data acquisition module 240 is used to obtain the current number of available CPU cores, current CPU load and remaining heap memory of the target device; the target device is the device that executes the object list data splitting method.

[0079] The data calculation module 250 is configured to calculate a parallel feasibility score of the target object list data according to the current number of available CPU cores, the current CPU load, and the remaining heap memory of the target device.

[0080] The data determination module 260 is configured to determine the remaining heap memory of the target device and the parallel feasibility score of the target object list data as the resource availability information of the target object list data.

[0081] Furthermore, when the strategy determination module 220 is used to determine the target splitting strategy of the target object list data based on the data feature information, business rule information and resource availability information of the target object list data, the strategy determination module 220 is specifically used to: determining whether the target object list data has a nested structure according to data characteristic information of the target object list data; the data characteristic information includes a list length and a list structure type of the target object list data; If so, determining a target splitting strategy for the target object list data based on the parallel feasibility score of the target object list data, the list length, and the remaining heap memory of the target device; If not, a target splitting strategy for the target object list data is determined based on the parallel feasibility score, list length, and business rule information of the target object list data.

[0082] Furthermore, when the strategy determination module 220 is used to determine the target splitting strategy of the target object list data based on the parallel feasibility score of the target object list data, the list length, and the remaining heap memory of the target device, the strategy determination module 220 is specifically used to: If the parallel feasibility score is greater than a preset feasibility score threshold, and the list length is greater than a preset first list length threshold, determining that the target splitting strategy includes a parallel splitting strategy and a recursive splitting strategy; If the parallel feasibility score is less than or equal to the feasibility score threshold or the list length is less than or equal to the first list length threshold, and the remaining heap memory of the target device is greater than a preset remaining heap memory threshold, determining that the target splitting strategy is a recursive splitting strategy; If the parallel feasibility score is less than or equal to the feasibility score threshold or the list length is less than or equal to the first list length threshold, and the remaining heap memory of the target device is less than or equal to the remaining heap memory threshold, the target splitting strategy is determined to be an iterative simulation recursive splitting strategy.

[0083] Furthermore, when the strategy determination module 220 is used to determine the target splitting strategy of the target object list data based on the parallel feasibility score, list length and business rule information of the target object list data, the strategy determination module 220 is specifically used to: If the list length is greater than a preset second list length threshold, and the parallel feasibility score is greater than the feasibility score threshold, determining that the target splitting strategy is a parallel splitting strategy; If the list length is greater than the second list length threshold and the parallel feasibility score is less than or equal to the feasibility score threshold and the number of business rules in the business rule information is greater than or equal to a preset business rule number threshold, or if the list length is less than or equal to the second list length threshold and the number of business rules is greater than or equal to the business rule number threshold, then determining that the target splitting strategy is a dynamic splitting strategy; If the list length is greater than the second list length threshold and the parallel feasibility score is less than or equal to the feasibility score threshold and the number of business rules is less than the business rule number threshold, or the list length is less than or equal to the second list length threshold and the number of business rules is less than the business rule number threshold, the target splitting strategy is determined to be the basic splitting strategy.

[0084] Furthermore, when the target splitting strategy includes a parallel splitting strategy, the data splitting module 230 is used to split the target object list data according to the target splitting strategy to obtain the target splitting result of the target object list data, the data splitting module 230 is specifically used to: Determining the number of split tasks of the parallel splitting strategy according to the number of available CPU cores of the target device and the list length of the target object list data; Create a thread pool with the same number of threads as the number of split tasks, and divide the target object list data into multiple subtasks according to the number of split tasks; Splitting the multiple subtasks in parallel according to the threads in the thread pool to obtain a splitting result for each subtask; The splitting results of each of the subtasks are collected and merged to obtain a target splitting result of the target object list data.

[0085] Furthermore, when the target splitting strategy includes a recursive splitting strategy, the data splitting module 230 is used to split the target object list data according to the target splitting strategy to obtain the target splitting result of the target object list data, and the data splitting module 230 is specifically used to: Traversing each element of the target object list data to determine a list structure type of each element; the list structure type includes a nested structure and a non-nested structure; For any element of the target object list data, if the list structure type of the element is a list structure in a nested structure, traverse each child element in the list structure of the element and recursively call the recursive splitting strategy for each child element; If the list structure type of the element is a mapping structure in a nested structure, traversing each value in the mapping structure of the element, and recursively calling the recursive splitting strategy for each value; If the list structure type of the element is a non-nested structure, the elements are matched according to the business rules in the target object list data business rule information, and the elements that meet the business rules are added to the sublist corresponding to the business rules; The sublists obtained by the recursive splitting are merged to obtain the target splitting result of the target object list data.

[0086] An embodiment of the present application provides a device for splitting object list data, comprising: a data acquisition module for acquiring target object list data to be split and extracting data feature information, business rule information, and resource availability information of the target object list data; a strategy determination module for determining a target splitting strategy for the target object list data based on the data feature information, business rule information, and resource availability information of the target object list data; and a data splitting module for splitting the target object list data according to the target splitting strategy to obtain a target splitting result for the target object list data. Thus, by analyzing the data feature information, business rule information, and resource availability information of the object list data, a suitable splitting strategy can be matched to the object list data, thereby improving the efficiency of object list data splitting.

[0087] Based on the same application idea, please refer to Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown, the electronic device 400 includes a processor 410 , a memory 420 and a bus 430 .

[0088] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 through the bus 430. The machine-readable instructions are executed by the processor 410 to execute the steps of the object list data splitting method provided in the above embodiment. The specific implementation method can be found in the method embodiment and will not be repeated here.

[0089] Based on the same application concept, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, the steps of the object list data splitting method provided in the above embodiment are executed. The specific implementation method can be found in the method embodiment and will not be repeated here.

[0090] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0091] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0092] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0093] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0094] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0095] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and are not to be understood as indicating or implying relative importance.

[0096] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present application, or make equivalent replacements for some of the technical features thereof. However, these modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.< / bean> < / t> < / t> < / t> < / bean> < / t> < / bean> < / bean> < / bean>

Claims

1. A method for splitting object list data, characterized in that: The method comprises: Acquire target object list data to be split, and extract data feature information, business rule information, and resource availability information of the target object list data; Determining a target splitting strategy for the target object list data based on data feature information, business rule information, and resource availability information of the target object list data; The target object list data is split according to the target splitting strategy to obtain a target splitting result of the target object list data.

2. The object list data splitting method according to claim 1, characterized in that: The resource availability information of the target object list data is extracted according to the following steps: Obtaining the current number of available CPU cores, current CPU load, and remaining heap memory of a target device; the target device is a device that executes the object list data splitting method; Calculating a parallel feasibility score for the target object list data based on the current number of available CPU cores, the current CPU load, and the remaining heap memory of the target device; The remaining heap memory of the target device and the parallel feasibility score of the target object list data are determined as resource availability information of the target object list data.

3. The object list data splitting method according to claim 2, characterized in that: The determining of the target splitting strategy of the target object list data based on the data feature information, business rule information, and resource availability information of the target object list data includes: determining whether the target object list data has a nested structure according to data characteristic information of the target object list data; the data characteristic information includes a list length and a list structure type of the target object list data; If so, determining a target splitting strategy for the target object list data based on the parallel feasibility score of the target object list data, the list length, and the remaining heap memory of the target device; If not, a target splitting strategy for the target object list data is determined based on the parallel feasibility score, list length, and business rule information of the target object list data.

4. The object list data splitting method according to claim 3, characterized in that: The determining of a target splitting strategy for the target object list data based on the parallel feasibility score of the target object list data, the list length, and the remaining heap memory of the target device includes: If the parallel feasibility score is greater than a preset feasibility score threshold, and the list length is greater than a preset first list length threshold, determining that the target splitting strategy includes a parallel splitting strategy and a recursive splitting strategy; If the parallel feasibility score is less than or equal to the feasibility score threshold or the list length is less than or equal to the first list length threshold, and the remaining heap memory of the target device is greater than a preset remaining heap memory threshold, determining that the target splitting strategy is a recursive splitting strategy; If the parallel feasibility score is less than or equal to the feasibility score threshold or the list length is less than or equal to the first list length threshold, and the remaining heap memory of the target device is less than or equal to the remaining heap memory threshold, the target splitting strategy is determined to be an iterative simulation recursive splitting strategy.

5. The object list data splitting method according to claim 4, characterized in that: The determining of a target splitting strategy for the target object list data based on the parallel feasibility score, list length, and business rule information of the target object list data includes: If the list length is greater than a preset second list length threshold, and the parallel feasibility score is greater than the feasibility score threshold, determining that the target splitting strategy is a parallel splitting strategy; If the list length is greater than the second list length threshold and the parallel feasibility score is less than or equal to the feasibility score threshold and the number of business rules in the business rule information is greater than or equal to a preset business rule number threshold, or if the list length is less than or equal to the second list length threshold and the number of business rules is greater than or equal to the business rule number threshold, then determining that the target splitting strategy is a dynamic splitting strategy; If the list length is greater than the second list length threshold and the parallel feasibility score is less than or equal to the feasibility score threshold and the number of business rules is less than the business rule number threshold, or the list length is less than or equal to the second list length threshold and the number of business rules is less than the business rule number threshold, the target splitting strategy is determined to be the basic splitting strategy.

6. The object list data splitting method according to claim 2, characterized in that: When the target splitting strategy includes a parallel splitting strategy, splitting the target object list data according to the target splitting strategy to obtain a target splitting result of the target object list data includes: Determining the number of split tasks of the parallel splitting strategy according to the number of available CPU cores of the target device and the list length of the target object list data; Create a thread pool with the same number of threads as the number of split tasks, and divide the target object list data into multiple subtasks according to the number of split tasks; Splitting the multiple subtasks in parallel according to the threads in the thread pool to obtain a splitting result for each subtask; The splitting results of each of the subtasks are collected and merged to obtain a target splitting result of the target object list data.

7. The object list data splitting method according to claim 1, characterized in that: When the target splitting strategy includes a recursive splitting strategy, splitting the target object list data according to the target splitting strategy to obtain a target splitting result of the target object list data includes: Traversing each element of the target object list data to determine a list structure type of each element; the list structure type includes a nested structure and a non-nested structure; For any element of the target object list data, if the list structure type of the element is a list structure in a nested structure, traverse each child element in the list structure of the element and recursively call the recursive splitting strategy for each child element; If the list structure type of the element is a mapping structure in a nested structure, traversing each value in the mapping structure of the element and recursively calling the recursive splitting strategy for each value; If the list structure type of the element is a non-nested structure, the elements are matched according to the business rules in the target object list data business rule information, and the elements that meet the business rules are added to the sublist corresponding to the business rules; The sublists obtained by the recursive splitting are merged to obtain the target splitting result of the target object list data.

8. A device for splitting object list data, characterized in that: The object list data splitting device includes: A data acquisition module is used to acquire target object list data to be split, and extract data feature information, business rule information and resource availability information of the target object list data; a strategy determination module, configured to determine a target splitting strategy for the target object list data based on data feature information, business rule information, and resource availability information of the target object list data; The data splitting module is used to split the target object list data according to the target splitting strategy to obtain a target splitting result of the target object list data.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and the machine-readable instructions are executed by the processor to execute the steps of the object list data splitting method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the object list data splitting method according to any one of claims 1 to 7 are executed.

Images