Splitting method and device of object list data, electronic equipment and storage medium

Abstract

The application provides a splitting method and device of object list data, an electronic device and a storage medium, wherein the splitting method of object list data comprises the following steps: obtaining target object list data to be split, and extracting data characteristic information, service rule information and resource availability information of the target object list data; determining a target splitting strategy of the target object list data based on the data characteristic information, the service rule information and the resource availability information of the target object list data; and splitting the target object list data according to the target splitting strategy to obtain a target splitting result of the target object list data. In this way, by analyzing the data characteristic information, the service rule information and the resource availability information of the object list data, an appropriate splitting strategy can be matched for the object list data, and the splitting efficiency of the object list data is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a splitting method and device of object list data, an electronic device and a storage medium. BACKGROUND

[0002] The object list data refers to a collection containing multiple objects, wherein each object has specific attributes and structures, for example, List <bean>Data. To efficiently process these data, it is often necessary to split them into multiple sub-lists.

[0003] Existing object list data splitting methods usually require developers to manually write loops and conditional judgment logic to process each object one by one and assign them to different sub-lists. In the face of large-scale data or complex structure scenarios, this manual splitting method is code-intensive and prone to errors, resulting in low efficiency of object list data splitting. SUMMARY

[0004] Therefore, the embodiments of the present application provide at least an object list data splitting method, device, electronic equipment and storage medium. By analyzing the data feature information, business rule information and resource availability information of the object list data, the object list data can be matched with an appropriate splitting strategy, and the efficiency of object list data splitting is improved.

[0005] The present application mainly includes the following aspects:

[0006] In a first aspect, the embodiments of the present application provide an object list data splitting method, which comprises:

[0007] Obtain target object list data to be split, and extract data feature information, business rule information and resource availability information of the target object list data;

[0008] Determine a target splitting strategy for the target object list data based on the data feature information, business rule information and resource availability information of the target object list data;

[0009] Split the target object list data according to the target splitting strategy to obtain a target splitting result of the target object list data.

[0010] In a second aspect, the embodiments of the present application also provide an object list data splitting device, which comprises:

[0011] A data acquisition module is configured to obtain target object list data to be split, and extract data feature information, business rule information and resource availability information of the target object list data;

[0012] A strategy determination module is configured to determine a target splitting strategy for the target object list data based on the data feature information, business rule information and resource availability information of the target object list data;

[0013] A data splitting module is configured to split the target object list data according to the target splitting strategy to obtain a target splitting result of the target object list data.

[0014] In a third aspect, the embodiments of the present application further provide an electronic device, comprising a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor communicates with the memory through the bus, and the machine readable instructions are executed by the processor to perform the steps of the object list data splitting method as described above.

[0015] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the object list data splitting method as described above.

[0016] The object list data splitting method, device, electronic device and storage medium provided by the embodiments of the present application, wherein the object list data splitting method comprises: obtaining target object list data to be split, and extracting data feature information, business rule information and resource availability information of the target object list data; determining a target splitting strategy of the target object list data based on the data feature information, the business rule information and the resource availability information of the target object list data; and splitting the target object list data according to the target splitting strategy to obtain a target splitting result of the target object list data. In this way, by analyzing the data feature information, the business rule information and the resource availability information of the object list data, an appropriate splitting strategy can be matched for the object list data, and the splitting efficiency of the object list data is improved.

[0017] In order to make the above objectives, characteristics and advantages of the present application more apparent and comprehensible, the following will specifically describe preferred embodiments in combination with the accompanying drawings, and make a detailed description as follows. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments, and it should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation to the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0019] Figure 1 A flow chart of an object list data splitting method provided by the embodiments of the present application is shown;

[0020] Figure 2 One of the functional module diagrams of an object list data splitting device provided by the embodiments of the present application is shown;

[0021] Figure 3 The second functional module diagram of an object list data splitting device provided by the embodiments of the present application is shown.

[0022] Figure 4 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] To make the objectives, technical solutions, and superiorities of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0024] To make the objectives, technical solutions, and superiorities of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0025] Please refer to Figure 1 , Figure 1 A flowchart of a splitting method of object list data provided by an embodiment of the present application is shown. As Figure 1 shown, the splitting method of object list data provided by the embodiment of the present application includes the following steps:

[0026] S101, obtaining target object list data to be split, and extracting data feature information, business rule information, and resource availability information of the target object list data.

[0027] Here, the target object list data to be split is first obtained. The target object list data refers to a collection containing multiple objects, each object having specific attributes and structure. These data are usually derived from business systems, databases, or data files, for subsequent processing and analysis. In the embodiment of the present application, the target object list data is a List <bean>Data. To efficiently process these data, three types of information related to splitting need to be extracted from the target object list data, namely data characteristic information, business rule information, and resource availability information. Among them, data characteristic information includes the list length and list structure type of the target object list data, which is used to describe the scale and composition structure of the data. Business rule information includes at least one business rule, i.e., the splitting condition of the target object list data, which is used to guide the specific way of data splitting. Resource availability information includes the current CPU available core number, current CPU load and remaining heap memory of the target device used for splitting the target object list data, which is used to assess whether the system resources support parallel processing and other operations. By extracting these information, comprehensive support basis can be provided for subsequent splitting strategy decision.

[0028] S102, determining a target splitting strategy of the target object list data based on the data characteristic information, business rule information and resource availability information of the target object list data.

[0029] Here, after obtaining the target object list data and its related information, a target splitting strategy is determined based on the data characteristic information, business rule information and resource availability information of the target object list data. This process considers data characteristics, business requirements and system resources to select the most suitable splitting method. By analyzing the scale, structure, business rules and availability of system resources of the data, the most suitable splitting strategy for the target object list data can be dynamically determined to ensure that the splitting process is both efficient and meets business requirements, i.e., improving the splitting efficiency and ensuring the accuracy of the splitting.

[0030] S103, splitting the target object list data according to the target splitting strategy to obtain a target splitting result of the target object list data.

[0031] Here, according to the determined target splitting strategy, the target object list data is executed for specific splitting operation. This process divides the target object list data into multiple sub-lists, each containing a set of objects that meet certain conditions. In this way, large-scale or complex-structure data can be decomposed into smaller pieces that are easier to process, thereby improving the efficiency and flexibility of data processing. The final target splitting result of the target object list data is obtained.

[0032] Further, the resource availability information of the target object list data is extracted according to the following steps:

[0033] Step a1, obtaining the current CPU available core number, current CPU load and remaining heap memory of the target device; the target device is a device that executes the splitting method of the object list data.

[0034] Here, the target device refers to the computer device that actually runs the object list data splitting method. Through the system interface or monitoring tool, the current CPU available core number of the device is obtained using the "Runtime.getRuntime().availableProcessors()" method , the current CPU load (typically a value between 0 and 1, representing the current CPU usage) and the remaining heap memory (in MB). These information reflect the resource status of the target device when performing the splitting operation.

[0035] Step a2, according to the current CPU available core number, current CPU load and remaining heap memory of the target device, calculate the parallel feasibility score of the target object list data.

[0036] Here, the parallel feasibility score is a comprehensive indicator to evaluate whether the target device is suitable for parallel processing. The score can be calculated by the following formula:

[0037] .

[0038] The formula takes into account the load of the CPU , the available core number and the remaining heap memory to determine the parallel processing capability of the target device in the current state.

[0039] Step a3, determine the remaining heap memory of the target device and the parallel feasibility score of the target object list data as the resource availability information of the target object list data.

[0040] Here, the resource availability information of the target object list data includes two key indicators: the remaining heap memory of the target device and the parallel feasibility score . These two indicators together reflect the resource status and parallel processing capability of the target device when performing the object list data splitting operation. By determining these two indicators as resource availability information, important reference can be provided for subsequent splitting strategy decision.

[0041] Further, the target splitting strategy of the target object list data is determined based on the data feature information, business rule information and resource availability information of the target object list data, including:

[0042] Step b1: Determine whether the target object list data has a nested structure based on the data feature information of the target object list data; the data feature information includes the list length and list structure type of the target object list data.

[0043] Here, the list length in the data feature information The list structure type is used to evaluate the complexity of the list data of the target object. List length. The list structure type indicates the size of the data, while the list structure type describes the composition of the data, specifically whether nested sublists or mapping structures exist. By analyzing this information, it's possible to determine whether the data has a nested structure, thus providing a basis for subsequent selection of target splitting strategies.

[0044] Step b2, if yes, then based on the parallel feasibility score of the target object list data, the list length, and the remaining heap memory of the target device, determine the target splitting strategy for the target object list data.

[0045] Here, if the target object list data has a nested structure, the parallel feasibility score needs to be further considered. List length and the remaining heap memory of the target device Parallel feasibility score This reflects the parallel processing capability of the target device in its current state; the list length... Indicates the size of the data, and the remaining heap memory. This reflects the system's memory resource status. By combining this information, the most suitable partitioning strategy can be determined, such as whether to use parallel partitioning or recursive partitioning.

[0046] Step b3: If not, then based on the parallel feasibility score, list length, and business rule information of the target object list data, determine the target splitting strategy for the target object list data.

[0047] Here, if the target object list data does not have a nested structure, then the parallel feasibility score of the target object list data is mainly considered. List length And business rule information. The business rule information includes at least one business rule that guides how to split the data based on specific conditions. By comprehensively considering these factors, the most suitable splitting strategy can be determined, such as whether to use parallel splitting or dynamic splitting.

[0048] Further, determining the target splitting strategy for the target object list data based on the parallel feasibility score, list length, and remaining heap memory of the target device includes:

[0049] Step c1: If the parallel feasibility score is greater than a preset feasibility score threshold and the list length is greater than a preset first list length threshold, then the target splitting strategy is determined to include a parallel splitting strategy and a recursive splitting strategy.

[0050] Here, when the parallel feasibility score Greater than the preset feasibility score threshold And the length of the target object list data Greater than the preset first list length threshold This indicates that the target device has sufficient computing resources to process large-scale data, and the data volume is large enough to benefit from parallel processing. Therefore, the target partitioning strategy is determined to be a combination of a parallel partitioning strategy and a recursive partitioning strategy. This combined strategy can fully utilize the computing power of multi-core processors, while handling nested structures, thus improving partitioning efficiency. In this embodiment, the feasibility score threshold... The value is 0.7, which is the threshold for the length of the first list. The value is 10000.

[0051] Step c2: If the parallel feasibility score is less than or equal to the feasibility score threshold or the list length is less than or equal to the first list length threshold, and the remaining heap memory of the target device is greater than the preset remaining heap memory threshold, then the target splitting strategy is determined to be a recursive splitting strategy.

[0052] Here, when the parallel feasibility score Less than or equal to the feasibility score threshold or list length Less than or equal to the first list length threshold However, the remaining heap memory of the target device Greater than the preset remaining heap memory threshold This indicates that although the conditions for parallel processing are not met, the device has sufficient memory to handle nested structures. Therefore, the target splitting strategy is determined to be a recursive splitting strategy. The recursive splitting strategy is suitable for processing nested structures, enabling layer-by-layer data decomposition and ensuring efficient processing. In this embodiment, the remaining heap memory threshold... The value is 512MB.

[0053] Step c3: If the parallel feasibility score is less than or equal to the feasibility score threshold or the list length is less than or equal to the first list length threshold, and the remaining heap memory of the target device is less than or equal to the remaining heap memory threshold, then the target splitting strategy is determined to be an iterative simulation recursive splitting strategy.

[0054] Here, when the parallel feasibility score Less than or equal to the feasibility score threshold and the list length is less than or equal to a first list length threshold , while the remaining heap memory of the target device is less than or equal to a remaining heap memory threshold , it indicates that both the computing resource and the memory resource of the target device are limited. In this case, the target splitting strategy is determined as an iterative simulation recursive splitting strategy. This strategy simulates the recursive process in an iterative manner, avoiding the stack overflow problem caused by recursive calls, while saving memory, and is suitable for resource-limited environments.

[0055] Further, the target splitting strategy of the target object list data is determined based on the parallel feasibility score, the list length and the business rule information of the target object list data, including:

[0056] In step d1, if the list length is greater than a preset second list length threshold, and the parallel feasibility score is greater than the feasibility score threshold, the target splitting strategy is determined as a parallel splitting strategy.

[0057] Here, when the length of the target object list data is greater than a preset second list length threshold , and the parallel feasibility score is greater than a preset feasibility score threshold , it indicates that the data volume is large and the target device has sufficient computing resources to support parallel processing. Therefore, the target splitting strategy is determined as a parallel splitting strategy. The parallel splitting strategy can fully utilize the computing power of multi-core processors, significantly improving the splitting efficiency. In the embodiment of the present application, the value of the second list length threshold is 50000.

[0058] In step d2, if the list length is greater than the second list length threshold and the parallel feasibility score is less than or equal to the feasibility score threshold and the number of business rules of the business rule information is greater than or equal to a preset business rule number threshold, or the list length is less than or equal to the second list length threshold and the number of business rules is greater than or equal to the business rule number threshold, the target splitting strategy is determined as a dynamic splitting strategy.

[0059] Here, when the list length is greater than the second list length threshold , but the parallel feasibility score does not meet the parallel processing condition (i.e. ), and the number of business rules in the business rule information is greater than or equal to a preset business rule number threshold , or when the list length is less than or equal to the second list length threshold But the number of business rules Still greater than or equal to the number of business rules threshold , it is indicated that although the parallel processing condition is not met, the business rule is complex and needs to be handled flexibly. Therefore, the target splitting strategy is determined as a dynamic splitting strategy. The dynamic splitting strategy can dynamically adjust the splitting logic according to the business rule, and is suitable for complex business scenarios. Among them, the value of the number of business rules threshold is 2.

[0060] In the embodiment of the application, the core goal of the dynamic splitting strategy is to flexibly and efficiently process data, aiming to process the List <bean>Data is split. The core design is based on interfacing, allowing developers to implement "SplitCondition <t>Interface custom business rules correspond to the split conditions (such as by name attribute). By traversing all the split conditions, the condition matching is performed for each element, and the elements meeting the conditions are added to the corresponding sub-list, and finally List<List <bean>The target splitting result of the structure. The specific steps are as follows:

[0061] First, traverse the splitting conditions of all business rules of the target object list data (each splitting condition is an independent logical unit for judging whether an element meets a specific condition. For example, condition 1: "name = Zhang San", condition 2: "age = 18"). Then, for each element of the target object list data, check whether the element meets the above splitting conditions. If the condition is met (put the element that meets a certain condition into a sub-list, and put the other elements into another list), add the element to the corresponding sub-list, and finally obtain the target splitting result of the target object list data.

[0062] In step d3, if the list length is greater than the second list length threshold and the parallel feasibility score is less than or equal to the feasibility score threshold and the number of business rules is less than the number of business rule thresholds, or the list length is less than or equal to the second list length threshold and the number of business rules is less than the number of business rule thresholds, the target splitting strategy is determined to be a basic splitting strategy.

[0063] Here, when the list length is greater than the second list length threshold , but the parallel feasibility score does not meet the parallel processing condition (that is ), and the number of business rules is less than the number of business rule thresholds , or when the list length is less than or equal to the second list length threshold and the number of business rules is also less than the number of business rule thresholds , it indicates that the data amount of the target object list data is small and the business rules are simple, and no complex splitting logic is needed. Therefore, the target splitting strategy is determined to be a basic splitting strategy.

[0064] In the embodiments of the present application, the basic splitting strategy can adopt any one of the following splitting strategies:

[0065] Manual loop splitting, which uses a custom-written tool class to implement the splitting function of List, combines loop and condition judgment, and manually writes loop traversal List according to specific conditions or fixed size to split elements into multiple sub-lists.

[0066] Use the "ListUtils.partition()" method provided by Apache Commons Collections, or the "Lists.partition()" method provided by Guava, to split a List into multiple sublists of a fixed size.

[0067] Using the Stream API introduced in Java 8, the "Collectors.groupingBy()" or "Collectors.partitioningBy()" methods are used to conditionally split the List.

[0068] Furthermore, when the target splitting strategy includes a parallel splitting strategy, the step of splitting the target object list data according to the target splitting strategy to obtain the target splitting result of the target object list data includes:

[0069] Step e1: Determine the number of splitting tasks for the parallel splitting strategy based on the number of available CPU cores of the target device and the list length of the target object list data.

[0070] Here, the core objective of the parallel partitioning strategy is to split large-scale data into multiple sublists through task sharding, multi-threaded parallel processing, and result merging, and finally merge the results after all thread tasks have completed. The number of partitioned tasks... The determination is based on the number of available CPU cores of the target device. and the list length of the target object list data By dynamically adjusting the number of tasks to be split, the efficiency of parallel processing and the rational utilization of resources can be ensured.

[0071] Specifically, the number of tasks to be split is determined according to the following rules. :

[0072] like When the data volume of the target object list is extremely small, in order to avoid wasting resources, it is determined that... .

[0073] like At that time, the amount of data in the target object list is determined to be moderate. and Balanced, at this point the default sharding settings are used. That is, 1 core and 1 thread.

[0074] like If the target object list data is large, the number of tasks to be split needs to be dynamically expanded. In this case, set... ,in, To expand the coefficient, with the upper limit is rounded up by the constraint. For example, when =4, =40000, the expansion coefficient = 2 (because 40000 / (5000x4)=2), so it is determined that T =8.

[0075] In this way, by dynamically adjusting the number of split tasks, both excessive fragmentation and excessive data allocation caused by single-thread blocking are avoided, ensuring the efficiency of parallel processing and the rationality of resource utilization.

[0076] Specifically, the input data List <t>The input data is divided into multiple sub-tasks, each of which is processed by a thread. The sharding strategy can dynamically adjust the number of shards according to the size of the thread pool. For example, if the number of threads in the thread pool is 4, the input data is divided into 4 sub-lists.

[0077] Step e2, create a thread pool with the number of threads equal to the number of split tasks, and divide the target object list data into multiple sub-tasks according to the number of split tasks.

[0078] Here, first, use ExecutorService to create a fixed-size thread pool with the number of threads equal to the number of split tasks, which can be represented as "ExecutorService executor = Executors.newFixedThreadPool(threadPoolSize)"; where "threadPoolSize" is the number of threads in the thread pool. The use of thread pool can effectively manage thread resources and improve the efficiency of parallel processing. Then, divide the target object list data into multiple sub-tasks according to the number of split tasks, ensuring that each sub-task contains approximately equal amount of data.

[0079] Step e3, according to the threads in the thread pool, parallel split the multiple sub-tasks to get the split result of each sub-task.

[0080] Here, use the threads in the thread pool to process each sub-task in parallel. Each thread independently performs the split operation of the sub-task, thereby fully utilizing the computing power of multi-core processors. Specifically, you can use "Future<List<List <t>The thread pool is managed by >>future = executor.submit(() ->splitSubList(subList, conditions)) and the subtasks are submitted for parallel processing.

[0081] Step e4, collecting the split results of each subtask and merging to obtain the target split result of the target object list data.

[0082] Here, the split results of all subtasks are collected and merged to obtain the final target split result. This can be achieved by waiting for all subtasks to complete and collecting their results. Finally, the split results of all subtasks are merged into a list of lists, where each sub-list contains elements that meet certain conditions. In the embodiment of the present application, the "Future" object is used to collect the split results of each thread "List<List <t>>subResult = future.get();result.addAll(subResult);”, wait for all thread tasks to be executed and close the thread pool (“executor.shutdown()”); and combine all sub-threads into the corresponding sub-list.

[0083] Further, when the target splitting strategy includes a recursive splitting strategy, the splitting of the target object list data according to the target splitting strategy to obtain a target splitting result of the target object list data includes:

[0084] Step f1, traversing each element of the target object list data to determine the list structure type of each element; the list structure type includes a nested structure and a non-nested structure.

[0085] Here, the core goal of the recursive splitting strategy is to traverse the nested data structure in depth and split the normal objects (such as Beans) according to the splitting condition (such as attribute value "name" splitting, entering the same sub-list for "name" consistency). For List and Map (element type identification is determined by the "instanceof" keyword, such as judging the List type: the current element "instanceof List"), the role of the recursive splitting strategy is to continue to traverse its sub-elements, rather than directly splitting them. Based on this recursive splitting, the input data is traversed and split according to the condition. Specifically, it is necessary to first traverse each element in the target object list data to determine the list structure type of each element. The list structure type includes a nested structure and a non-nested structure, and the list structure type can be a list structure List in the nested structure (for example, an object list containing another object list), a mapping structure Map (for example, an object mapping containing another object list or mapping), or a non-nested structure, i.e. a normal object such as a Bean. By identifying these list structure types, it can be determined how to further process each element.

[0086] Step f2, for any element of the target object list data, if the list structure type of the element is a list structure in the nested structure, then traverse each sub-element in the list structure of the element and recursively call the recursive splitting strategy for each sub-element.

[0087] Here, if the current element is a list structure List, it is necessary to further traverse each sub-element in the list. For each sub-element, the recursive splitting strategy is recursively called to handle possible deeper nested structures, wherein the sub-element Bean in the recursion is also split by the splitting condition (such as attribute value "name"), ensuring comprehensive processing of multi-level nested structures.

[0088] Step f3, if the list structure type of the element is a map structure in a nested structure, then traverse each value in the map structure of the element and recursively call the recursive splitting strategy on each value.

[0089] Here, if the current element is a map structure Map, each value in the map structure Map needs to be traversed. For each value, the recursive splitting strategy is recursively called to handle the possible nested structure. The recursive child element Bean is also split by the splitting condition (such as attribute value "name") to ensure that the nested data in the map structure Map is processed deeply.

[0090] Step f4, if the list structure type of the element is a non-nested structure, then match the element according to the business rule in the target object list data business rule information, and add the element meeting the business rule to the corresponding business rule sub-list.

[0091] Here, if the current element is a normal object (such as Bean), i.e. a non-nested structure, then match the element according to the splitting condition (such as attribute value "name") of the business rule in the business rule information. If the element meets the splitting condition of a certain business rule, it is added to the corresponding sub-list.

[0092] Step f5, merge the sub-lists obtained by recursive splitting to obtain the target splitting result of the target object list data.

[0093] Here, all the sub-lists obtained in the recursive splitting process are merged to finally obtain the target splitting result of the target object list data. This step ensures that all the processed data is integrated together to form a List<List <bean>Final target split result of the structure, wherein each sublist contains elements satisfying a certain split condition.

[0094] The object list data splitting method provided by the embodiment of the present application comprises the following steps: obtaining target object list data to be split, and extracting data characteristic information, business rule information and resource availability information of the target object list data; determining a target splitting strategy of the target object list data based on the data characteristic information, the business rule information and the resource availability information of the target object list data; and splitting the target object list data according to the target splitting strategy to obtain a target splitting result of the target object list data. In this way, by analyzing the data characteristic information, the business rule information and the resource availability information of the object list data, an appropriate splitting strategy can be matched for the object list data, and the splitting efficiency of the object list data is improved.

[0095] Based on the same application concept, the embodiment of the present application also provides an object list data splitting device corresponding to the object list data splitting method provided by the above-mentioned embodiment. Since the principle of solving problems by the device in the embodiment of the present application is similar to the object list data splitting method of the above-mentioned embodiment of the present application, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described here.

[0096] Please refer to Figure 2 , Figure 2 One of the functional module diagrams of the object list data splitting device provided by the embodiment of the present application. As shown in Figure 2 , the object list data splitting device 200 comprises:

[0097] The data acquisition module 210 is configured to obtain target object list data to be split, and extract data characteristic information, business rule information and resource availability information of the target object list data.

[0098] The strategy determination module 220 is configured to determine a target splitting strategy of the target object list data based on the data characteristic information, the business rule information and the resource availability information of the target object list data.

[0099] The data splitting module 230 is configured to split the target object list data according to the target splitting strategy to obtain a target splitting result of the target object list data.

[0100] Further, please refer to Figure 3 , Figure 3 The second functional module diagram of the object list data splitting device provided by the embodiment of the present application. As shown in Figure 3 , the object list data splitting device 200 further comprises:

[0101] The device data obtaining module 240 is configured to obtain current CPU available core number, current CPU load and remaining heap memory of a target device; the target device is a device for executing the splitting method of the object list data.

[0102] The data calculating module 250 is configured to calculate parallel feasibility score of the target object list data according to the current CPU available core number, the current CPU load and the remaining heap memory of the target device.

[0103] The data determining module 260 is configured to determine the remaining heap memory of the target device and the parallel feasibility score of the target object list data as resource availability information of the target object list data.

[0104] Further, the policy determining module 220 is configured to:

[0105] determine whether the target object list data has a nested structure according to the data characteristic information of the target object list data; the data characteristic information comprises list length and list structure type of the target object list data;

[0106] if yes, determine the target splitting strategy of the target object list data based on the parallel feasibility score, the list length of the target object list data and the remaining heap memory of the target device;

[0107] if no, determine the target splitting strategy of the target object list data based on the parallel feasibility score, the list length of the target object list data and the service rule information.

[0108] Further, the policy determining module 220 is configured to:

[0109] if the parallel feasibility score is greater than a preset feasibility score threshold and the list length is greater than a preset first list length threshold, determine that the target splitting strategy comprises a parallel splitting strategy and a recursive splitting strategy;

[0110] if the parallel feasibility score is less than or equal to the feasibility score threshold or the list length is less than or equal to the first list length threshold, and the remaining heap memory of the target device is greater than a preset remaining heap memory threshold, determine that the target splitting strategy is a recursive splitting strategy;

[0111] If the parallel feasibility score is less than or equal to the feasibility score threshold or the list length is less than or equal to the first list length threshold, and the remaining heap memory of the target device is less than or equal to the remaining heap memory threshold, the target splitting strategy is determined as an iterative simulation recursive splitting strategy.

[0112] Further, when determining the target splitting strategy of the target object list data based on the parallel feasibility score, the list length and the service rule information of the target object list data, the policy determination module 220 is specifically configured to:

[0113] If the list length is greater than a preset second list length threshold, and the parallel feasibility score is greater than the feasibility score threshold, the target splitting strategy is determined as a parallel splitting strategy.

[0114] If the list length is greater than the second list length threshold and the parallel feasibility score is less than or equal to the feasibility score threshold, and the number of service rules of the service rule information is greater than or equal to a preset service rule number threshold, or the list length is less than or equal to the second list length threshold and the number of service rules is greater than or equal to the service rule number threshold, the target splitting strategy is determined as a dynamic splitting strategy.

[0115] If the list length is greater than the second list length threshold and the parallel feasibility score is less than or equal to the feasibility score threshold, and the number of service rules is less than the service rule number threshold, or the list length is less than or equal to the second list length threshold and the number of service rules is less than the service rule number threshold, the target splitting strategy is determined as a basic splitting strategy.

[0116] Further, when the target splitting strategy includes a parallel splitting strategy, the data splitting module 230 is specifically configured to:

[0117] determine the number of splitting tasks of the parallel splitting strategy according to the number of available CPU cores of the target device and the list length of the target object list data;

[0118] create a thread pool with a number of threads equal to the number of splitting tasks, and divide the target object list data into a plurality of subtasks according to the number of splitting tasks;

[0119] parallelize the plurality of subtasks according to the threads in the thread pool to obtain a splitting result of each subtask;

[0120] Collect the split results of each of the subtasks, and merge to obtain a target split result of the target object list data.

[0121] Further, when the target split strategy includes a recursive split strategy, the data split module 230 is specifically configured to:

[0122] traverse each element of the target object list data, and determine a list structure type of each element; the list structure type includes a nested structure and a non-nested structure;

[0123] For any element of the target object list data, if the list structure type of the element is a list structure in the nested structure, each sub-element in the list structure of the element is traversed, and the recursive split strategy is recursively called for each sub-element;

[0124] if the list structure type of the element is a mapping structure in the nested structure, each value in the mapping structure of the element is traversed, and the recursive split strategy is recursively called for each value;

[0125] if the list structure type of the element is a non-nested structure, the element is matched according to the business rule in the business rule information of the target object list data, and the element meeting the business rule is added to a sub-list corresponding to the business rule;

[0126] merge each sub-list obtained by recursive splitting to obtain a target split result of the target object list data.

[0127] The object list data splitting device provided by the embodiment of the present application comprises a data acquisition module, a strategy determination module, and a data split module. The data acquisition module is configured to acquire target object list data to be split, and extract data characteristic information, business rule information, and resource availability information of the target object list data. The strategy determination module is configured to determine a target split strategy of the target object list data based on the data characteristic information, the business rule information, and the resource availability information of the target object list data. The data split module is configured to split the target object list data according to the target split strategy, and obtain a target split result of the target object list data. In this way, by analyzing the data characteristic information, the business rule information, and the resource availability information of the object list data, an appropriate split strategy can be matched for the object list data, and the splitting efficiency of the object list data is improved.

[0128] Based on the same application concept, please refer to Figure 4 , Figure 4 A structural schematic diagram of an electronic device provided by the embodiment of the present application. As shown in Figure 4 As shown in FIG. 4, the electronic device 400 includes a processor 410, a memory 420 and a bus 430.

[0129] The memory 420 stores machine readable instructions executable by the processor 410, and when the electronic device 400 is running, the processor 410 communicates with the memory 420 through the bus 430. The machine readable instructions are executed by the processor 410 to perform the steps of the object list data splitting method provided in the above embodiments. For details, refer to the method embodiments, which will not be repeated here.

[0130] Based on the same application concept, the embodiments of the present application also provide a computer readable storage medium, which stores a computer program. When the computer program is run by a processor, the steps of the object list data splitting method provided in the above embodiments are performed. For details, refer to the method embodiments, which will not be repeated here.

[0131] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0132] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented by other means. The device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interface, device or unit, which can be electrical, mechanical or other forms.

[0133] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the present embodiment.

[0134] In addition, each functional unit in the embodiments provided in the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0135] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions of the present application can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0136] It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. In addition, the terms "first", "second", "third" and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0137] Finally, it should be noted that the above-described embodiments are only specific implementations of the present application, which are used to illustrate the technical solutions of the present application, but not to limit them. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some of the technical features. These modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.< / bean> < / t> < / t> < / t> < / bean> < / t> < / bean> < / bean> < / bean>

Claims

1. A method of splitting object list data, characterized by, The method comprises: acquiring target object list data to be split, and extracting data feature information, business rule information and resource availability information of the target object list data; determining a target splitting strategy of the target object list data based on the data feature information, the business rule information and the resource availability information of the target object list data; splitting the target object list data according to the target splitting strategy to obtain a target splitting result of the target object list data; when the target splitting strategy comprises a recursive splitting strategy, the splitting of the target object list data according to the target splitting strategy to obtain the target splitting result of the target object list data comprises: traversing each element of the target object list data to determine a list structure type of each element; the list structure type comprises a nested structure and a non-nested structure; for any element of the target object list data, if the list structure type of the element is a list structure in the nested structure, traversing each sub-element in the list structure of the element and recursively calling the recursive splitting strategy on each sub-element; if the list structure type of the element is a mapping structure in the nested structure, traversing each value in the mapping structure of the element and recursively calling the recursive splitting strategy on each value; if the list structure type of the element is a non-nested structure, matching the element according to a business rule in the business rule information of the target object list data, and adding the element meeting the business rule to a sub-list corresponding to the business rule; merging each sub-list obtained by recursive splitting to obtain the target splitting result of the target object list data.

2. The method of splitting object list data according to claim 1, wherein, The resource availability information of the target object list data is extracted according to the following steps: acquiring the current CPU available core number, the current CPU load and the remaining heap memory of a target device; the target device is a device for executing the splitting method of the object list data; calculating the parallel feasibility score of the target object list data according to the current CPU available core number, the current CPU load and the remaining heap memory of the target device; determining the resource availability information of the target object list data based on the remaining heap memory of the target device and the parallel feasibility score of the target object list data.

3. The method of splitting object list data according to claim 2, wherein, The determination of the target splitting strategy of the target object list data based on the data feature information, the business rule information and the resource availability information of the target object list data comprises: judging whether the target object list data has a nested structure according to the data feature information of the target object list data; the data feature information comprises the list length and the list structure type of the target object list data; if yes, determining the target splitting strategy of the target object list data based on the parallel feasibility score, the list length of the target object list data and the remaining heap memory of the target device; if no, determining the target splitting strategy of the target object list data based on the parallel feasibility score, the list length and the business rule information of the target object list data.

4. The method of splitting object list data according to claim 3, wherein, The target splitting strategy of the target object list data is determined based on the parallel feasibility score, the list length and the remaining heap memory of the target device, and the target splitting strategy of the target object list data comprises: If the parallel feasibility score is greater than a preset feasibility score threshold and the list length is greater than a preset first list length threshold, the target splitting strategy comprises a parallel splitting strategy and a recursive splitting strategy; If the parallel feasibility score is less than or equal to the feasibility score threshold or the list length is less than or equal to the first list length threshold, and the remaining heap memory of the target device is greater than a preset remaining heap memory threshold, the target splitting strategy is determined as a recursive splitting strategy; If the parallel feasibility score is less than or equal to the feasibility score threshold or the list length is less than or equal to the first list length threshold, and the remaining heap memory of the target device is less than or equal to the remaining heap memory threshold, the target splitting strategy is determined as an iterative simulation recursive splitting strategy.

5. The method of splitting object list data according to claim 4, wherein, The target splitting strategy of the target object list data is determined based on the parallel feasibility score, the list length and the service rule information, and the target splitting strategy of the target object list data comprises: If the list length is greater than a preset second list length threshold and the parallel feasibility score is greater than the feasibility score threshold, the target splitting strategy is determined as a parallel splitting strategy; If the list length is greater than the second list length threshold and the parallel feasibility score is less than or equal to the feasibility score threshold and the number of service rules of the service rule information is greater than or equal to a preset number of service rules threshold, or the list length is less than or equal to the second list length threshold and the number of service rules is greater than or equal to the number of service rules threshold, the target splitting strategy is determined as a dynamic splitting strategy; If the list length is greater than the second list length threshold and the parallel feasibility score is less than or equal to the feasibility score threshold and the number of service rules is less than the number of service rules threshold, or the list length is less than or equal to the second list length threshold and the number of service rules is less than the number of service rules threshold, the target splitting strategy is determined as a basic splitting strategy.

6. The method of splitting object list data according to claim 2, wherein, When the target splitting strategy comprises a parallel splitting strategy, the target object list data is split according to the target splitting strategy to obtain a target splitting result of the target object list data, and the method comprises: The number of splitting tasks of the parallel splitting strategy is determined according to the number of available CPU cores of the target device and the list length of the target object list data; A thread pool is created, in which the number of threads is equal to the number of splitting tasks, and the target object list data is divided into a plurality of subtasks according to the number of splitting tasks; The plurality of subtasks are split in parallel according to the threads in the thread pool to obtain a splitting result of each subtask; The splitting results of the subtasks are collected and combined to obtain the target splitting result of the target object list data.

7. An apparatus for splitting object list data, characterized by comprising: The object list data splitting device comprises: The data acquisition module is configured to acquire target object list data to be split, and extract data feature information, business rule information, and resource availability information of the target object list data. The policy determination module is configured to determine a target splitting policy of the target object list data based on the data feature information, the business rule information, and the resource availability information of the target object list data. The data splitting module is configured to split the target object list data according to the target splitting policy, to obtain a target splitting result of the target object list data. When the target splitting policy includes a recursive splitting policy, the data splitting module is specifically configured to: traverse each element of the target object list data, determine a list structure type of each element, the list structure type includes a nested structure and a non-nested structure, for any element of the target object list data, if the list structure type of the element is a list structure in the nested structure, traverse each sub-element in the list structure of the element, and recursively call the recursive splitting policy on each sub-element, if the list structure type of the element is a mapping structure in the nested structure, traverse each value in the mapping structure of the element, and recursively call the recursive splitting policy on each value, if the list structure type of the element is the non-nested structure, match the element according to a business rule in the business rule information of the target object list data, and add an element meeting the business rule to a sub-list corresponding to the business rule, and combine each sub-list obtained by recursive splitting to obtain the target splitting result of the target object list data.

8. An electronic device, comprising: The processor, the memory, and the bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, the machine readable instructions are executed by the processor to execute the steps of the object list data splitting method in any one of claims 1 to 6. The computer readable storage medium stores a computer program, the computer program is executed by the processor to execute the steps of the object list data splitting method in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, ​

Images