A high-performance, easy-to-expand unified search method based on Spring

Abstract

The application relates to the technical field of software development, and discloses a high-performance and easily-extensible unified search method based on Spring, which comprises the following steps: step one, defining a unified search interface, the search interface providing a unified programming model for all search services, the search interface comprising a search method, the search method receiving search parameters and returning search results; step two, in the process of starting the Spring container, automatically identifying and injecting all beans implementing the search interface in step one; step three, when the system receives a unified search request, the search center traverses all registered beans related to the search service; for each bean, the search method in step one is called to initiate a search task; and step four, the search tasks are processed in parallel through a thread pool, the search tasks are analyzed based on system running data and task data, and the size of the thread pool is dynamically adjusted according to the analysis result.

Description

Technical Field

[0001] This application relates to the field of software development technology, and in particular to a high-performance, easily extensible unified search method based on Spring. Background Technology

[0002] In the face of an era of information explosion and increasingly diversified search needs, providing users with a unified and efficient search portal and service has become an indispensable cornerstone for businesses and platforms today.

[0003] Traditional search systems often require customized search logic for different data sources and formats. Each time a new search scenario is introduced, the core search code needs to be adjusted. This not only leads to frequent adjustments to the core code and requires a large investment of manpower, but also affects the stability and scalability of the system. Therefore, how to provide a unified and efficient search solution and service while ensuring the stability and scalability of the system is the fundamental problem that this invention aims to solve. Summary of the Invention

[0004] To provide a unified and efficient search method and service while ensuring system stability and scalability, this application offers a high-performance, easily extensible unified search method based on Spring, employing the following technical solution: A high-performance, easily extensible unified search method based on Spring, including: Step 1: Define a unified search interface. The search interface provides a unified programming model for all search services. The search interface includes a search method that receives search parameters and returns search results. Step 2: During the Spring container startup process, automatically identify and inject all beans that implement the search interface described in Step 1; Step 3: When the system receives a unified search request, the search center iterates through all registered beans related to the search service; for each bean, it calls the search method in Step 1 to initiate a search task. Step 4: Process the search tasks in parallel using a thread pool, analyze the search tasks based on system operation data and task data, and dynamically adjust the size of the thread pool according to the analysis results. Step 5: After all search tasks are completed, obtain the search results set, aggregate the search results set, and return it to the user; The system operation data includes CPU utilization, memory utilization, and the current number of threads; The task data includes the search type of the task to be processed and historical data of the processed tasks; The process of analyzing a search task includes: Based on historical data of processed tasks, perform characteristic analysis on different search types, and predict the load of tasks to be processed based on the results of the characteristic analysis, the search type of tasks to be processed, and the waiting time of tasks to be processed. The thread pool size adjustment strategy is determined based on system operation data and the predicted load of pending tasks. The process of performing feature analysis on different search types includes: Extract the CPU usage time series Oc(t1), Oc(t1), ..., Oc(tn) and memory usage time series Om(t1), Om(t1), ..., Om(tn) for each processed task from the historical data of processed tasks; A CPU usage curve was fitted based on the CPU usage time series of all processed tasks. ; Calculate the CPU usage time series and CPU usage curve for each processed task. Calculate the sum of squared residuals Sc; calculate the standard deviation of the sum of squared residuals Sc for all processed tasks. ; A memory usage curve was fitted based on the memory usage time series of all processed tasks. ; Calculate the memory usage time series and memory usage curve for each processed task. Calculate the residual sum of squares Sm; calculate the standard deviation of the residual sum of squares Sm for all processed tasks. ; By equation: Calculate and obtain reference CPU usage curves for various task search characteristics. and reference memory usage curve ; Obtain the reference CPU usage curve corresponding to the search type of the task to be processed. and reference memory usage curve ; in, Standard deviation The corresponding reference threshold, This is the first fixed coefficient; Standard deviation The corresponding reference threshold, This is the second fixed coefficient.

[0005] Optionally, the process of predicting the workload of the task to be processed includes: Using the equations respectively: Calculate and obtain the predicted CPU load value and memory predicted load value Based on CPU predicted load value and memory predicted load value Predict the workload of tasks to be processed; Where m is the number of tasks to be processed, and j∈[1,m], This serves as a reference CPU usage curve for the characteristics of the j-th task to be processed. Indicates taking Maximum value This is a reference memory usage curve for the characteristics of the j-th task to be processed. Indicates taking Maximum value.

[0006] Optionally, the process of determining the thread pool size adjustment strategy includes: Through equations Calculate the required number of threads Q, and compare Q with the current number of threads Q0: If Q≥Q0, determine whether the system performance meets the requirements after increasing the number of threads: If the requirement is met, the thread pool size will be adjusted according to the required number of threads Q. If the conditions are not met, the thread pool size is determined based on system operating data and the difference between Q and Q0. If Q < Q0, determine whether to adjust the thread pool size based on system operating data and the difference between Q and Q0; in, This is a function that compares CPU predicted load value with the number of threads. A function to predict memory load value versus thread count. Indicates taking and The maximum value in.

[0007] Optionally, when Q≥Q0, the process of determining whether the system performance is satisfied after increasing the number of threads includes: Using the equations respectively: Calculate the remaining CPU percentage and remaining memory percentage CPU percentage and remaining memory percentage Compare with the corresponding thresholds respectively, when and If all values ​​are greater than the corresponding threshold, it is determined that the system performance can be satisfied after increasing the number of threads; otherwise, it is determined that the system performance cannot be satisfied after increasing the number of threads. in, This represents the current percentage of CPU idle time. Preset CPU percentage for single-threaded operation. This represents the current percentage of free memory. Preset memory allocation for single-threaded applications; When it is determined that increasing the number of threads will not meet system performance requirements, the process of determining the thread pool size includes: Through equations Calculate the thread pool size ; in, , Remaining CPU percentage and remaining memory percentage The corresponding threshold, The floor symbol is used for rounding down. Indicates taking and The minimum value in.

[0008] Optionally, when Q≥Q0, the process of determining whether to adjust the thread pool size based on system operating data and the difference between Q and Q0 includes: Through equations Calculate the thread pool size ; Where f(x) is a defined function, f(x) = x when x ≥ 0, otherwise f(x) = 0; , Remaining CPU percentage and remaining memory percentage The reference value, and satisfying > , > ; Indicates taking and The minimum value in.

[0009] Optionally, the search task can be encapsulated as a CompletableFuture object, and asynchronous operation management can be performed based on CompletableFuture in JAVA8.

[0010] In summary, this application includes at least one of the following beneficial technical effects: This invention allows for the easy incorporation of new data types and sources without altering the core code. By establishing an abstract data access layer and a highly modular processing engine, the system can automatically identify and adapt to diverse data formats, ensuring the stability of the core code. This means that if a new search scenario needs to be added, the development team can quickly expand the functionality without significantly modifying the core code. Furthermore, the unified search emphasizes performance optimization, utilizing multi-threaded concurrency technology and distributed computing resources to ensure rapid response to user queries even when processing massive amounts of data. This not only improves the user experience but also provides a solid foundation for handling high-concurrency scenarios. Attached Figure Description

[0011] Figure 1 This is a logical diagram of the high-performance, easily extensible unified search method based on Spring in this invention; Figure 2 This is a flowchart illustrating the steps of the high-performance, easily extensible unified search method based on Spring, as described in this invention. Detailed Implementation

[0012] The embodiments of this application are described in detail below, and examples of the embodiments are shown in the accompanying drawings.

[0013] In the description of this specification, the references to "certain embodiments," "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples" refer to specific features, structures, materials, or characteristics described in connection with the described embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0014] This application discloses a high-performance, easily extensible unified search method based on Spring, referring to... Figures 1-2As shown, the process includes: Step 1: Define a unified search interface. This interface provides a unified programming model for all search services. The interface includes a search method that receives search parameters and returns search results. This unified programming model ensures consistency and interoperability between different services. Step 2: During Spring container startup, automatically identify and inject all beans that implement the search interface described in Step 1. This process leverages Spring's dependency injection and auto-configuration features to automatically identify and register all beans implementing the interface in Step 1. This step requires no manual intervention; any subsequent services that need to integrate will directly access the relevant beans in the container. This greatly improves the convenience of service access and the scalability of the system. Step 3: When the system receives a unified search request, the search center iterates through all registered beans related to the search service. For each bean, it calls the search method from Step 1 to initiate a search task and encapsulates the search task as a CompletableFuture object. Step 4: The search task is processed in parallel using a thread pool, and the search task is analyzed based on system runtime data and task data. The size of the thread pool is dynamically adjusted according to the analysis results. This process maximizes the optimization of resource usage and response time. Simultaneously, using CompletableFuture in Java 8, asynchronous operations can be easily managed, and results are aggregated after all search tasks are completed. Step 5: After all search tasks are completed, the search result set is obtained, aggregated, and returned to the user. During the search process, the main thread waits for all search tasks to complete. If a search task fails, the system catches the exception and records the error information, while ensuring that other search tasks continue to execute. This improves the system's robustness. Once all tasks represented by CompletableFuture objects are completed, all results are retrieved using the CompletableFuture.allOf method and aggregated into a unified search result set. Finally, the system returns this aggregated result to the user, providing a comprehensive and consistent search experience. The unified search method in this embodiment allows for the easy inclusion of new data types and sources without altering the core code. By establishing an abstract data access layer and a highly modular processing engine, the system can automatically identify and adapt to diverse data formats, ensuring the stability of the core code. This allows the development team to quickly expand functionality without significant modifications to the core code should a new search scenario be added. Furthermore, unified search emphasizes performance optimization, utilizing multi-threaded concurrency technology and distributed computing resources to ensure rapid response to user queries even when processing massive amounts of data. This not only enhances the user experience but also provides a solid foundation for handling high-concurrency scenarios.

[0015] In one embodiment, a method for dynamically adjusting the thread pool size is provided, wherein system operation data includes CPU utilization, memory utilization, and the current number of threads; task data includes the search type of pending tasks and historical data of processed tasks; the process of analyzing search tasks includes: performing characteristic analysis on different search types based on the historical data of processed tasks; predicting the load of pending tasks based on the characteristic analysis results, the search type of pending tasks, and the waiting time of pending tasks; determining the thread pool size adjustment strategy based on system operation data and the predicted load of pending tasks; the process of performing characteristic analysis on different search types includes: extracting the CPU usage time series Oc(t1), Oc(t1), ..., Oc(tn) and memory usage time series Om(t1), Om(t1), ..., Om(tn) of each processed task from the historical data of processed tasks; and fitting a CPU usage curve based on the CPU usage time series of all processed tasks. ; Calculate the CPU usage time series and CPU usage curve for each processed task. The sum of squared residuals Sc; CPU usage curves at each point in the CPU usage time series. The residual reflects the relationship between this point and The degree of deviation is determined by calculating the CPU usage time series at all points and the CPU usage curve. The residual sum of squares, Sc, can be used to analyze the CPU usage curve for each task. The degree of deviation is judged, and then the standard deviation of the sum of squared residuals (Sc) of all processed tasks is calculated. This allows us to determine the discrete state of all processed tasks, facilitating subsequent calculations of CPU usage curves. Similarly, a memory usage curve is fitted based on the memory usage time series of all processed tasks. ; Calculate the memory usage time series and memory usage curve for each processed task. Calculate the residual sum of squares Sm; calculate the standard deviation of the residual sum of squares Sm for all processed tasks. ; By equation: Calculate and obtain reference CPU usage curves for various task search characteristics. and reference memory usage curve ; Obtain the reference CPU usage curve corresponding to the search type of the task to be processed. and reference memory usage curve ;in, Standard deviation The corresponding reference threshold, This is the first fixed coefficient; Standard deviation The corresponding reference threshold, As the second fixed coefficient, among the above parameters, the reference threshold and Based on the error settings present in the test data, when and If the value is less than the corresponding threshold, it means it is within the error range, and in this case, it can be directly... As a reference CPU usage curve As a reference memory usage curve, conversely, when and If it is less than the corresponding threshold, then... and Adjustments are made, the first fixed coefficient. and the second fixed coefficient Fitting settings based on test data: This process involves substituting a large amount of validated data to obtain the optimal first fixed coefficient. and the second fixed coefficient .

[0016] Through the above characteristic analysis process, we can better determine the load status of the tasks to be processed. That is, based on the characteristic analysis results, the search type of the tasks to be processed, and the waiting time of the tasks to be processed, we can predict the load of the tasks to be processed. This process includes: using the following equations respectively: , Calculate and obtain the predicted CPU load value and memory predicted load value Where m is the number of tasks to be processed, and j∈[1,m], This serves as a reference CPU usage curve for the characteristics of the j-th task to be processed. Indicates taking Maximum value This is a reference memory usage curve for the characteristics of the j-th task to be processed. Indicates taking Maximum value, therefore based on CPU predicted load value and memory predicted load value Predicting the load on tasks allows for determining the thread pool size based on the load status of those tasks. This process includes: using equations... The required number of threads Q is calculated, where, This is a function that compares CPU predicted load value with the number of threads. A function to predict memory load value versus thread count. and Based on empirical data showing the optimal number of threads for different CPU and memory loads, a corresponding relationship is established. Fitting this relationship yields the corresponding function. Therefore... and Both are piecewise functions. Indicates taking and The maximum value in the range is required, therefore the system needs to simultaneously meet the CPU and memory requirements, and thus the number of threads needs to satisfy both, hence the selection. and The maximum value in the range is taken as the required number of threads Q. The required number of threads Q is compared with the current number of threads Q0: if Q≥Q0, it is determined whether the system performance is satisfied after increasing the number of threads. This process includes: respectively through the equation: , Calculate the remaining CPU percentage and remaining memory percentage ,in, This represents the current percentage of CPU idle time. Preset CPU percentage for single-threaded operation. This represents the current percentage of free memory. Preset memory allocation for single-threaded applications; and Based on empirical data, set the CPU percentage. and remaining memory percentage The CPU percentage was compared with the corresponding thresholds. and remaining memory percentage The corresponding threshold is set based on the current system performance status and empirical data. That is, the system must operate normally under the corresponding threshold. Therefore, when... and If all values ​​are greater than the corresponding threshold, it is determined that increasing the number of threads will satisfy the system performance requirement; otherwise, it is determined that increasing the number of threads will not satisfy the system performance requirement. If the requirement is met, the thread pool size is adjusted according to the required number of threads Q. If the requirement is not met, the thread pool size is determined based on system operating data and the difference between Q and Q0. This process includes: using the equation... Calculate the thread pool size ;in, , Remaining CPU percentage and remaining memory percentage The corresponding threshold is set based on the current system performance status and empirical data. The floor symbol is used for rounding down. Indicates taking and The minimum value in the equation; if Q < Q0, determine whether to adjust the thread pool size based on system operating data and the difference between Q and Q0. This process includes: using the equation... Calculate the thread pool size Where f(x) is a defined function, f(x) = x when x ≥ 0, otherwise f(x) = 0; , Remaining CPU percentage and remaining memory percentage The reference value, and satisfying > , > ; Indicates taking and The minimum value in the above process can accurately determine the optimal number of threads based on the predicted workload of the tasks to be processed. At the same time, by rounding down and taking the minimum value, the frequency of thread number changes is reduced, thereby reducing the overhead on the system.

[0017] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A high-performance, easily extensible unified search method based on Spring, characterized in that, include: Step 1: Define a unified search interface. The search interface provides a unified programming model for all search services. The search interface includes a search method that receives search parameters and returns search results. Step 2: During the Spring container startup process, automatically identify and inject all beans that implement the search interface described in Step 1; Step 3: When the system receives a unified search request, the search center iterates through all registered beans related to the search service; for each bean, it calls the search method in Step 1 to initiate a search task. Step 4: Process the search tasks in parallel using a thread pool, analyze the search tasks based on system operation data and task data, and dynamically adjust the size of the thread pool according to the analysis results. Step 5: After all search tasks are completed, obtain the search results set, aggregate the search results set, and return it to the user; The system operation data includes CPU utilization, memory utilization, and the current number of threads; The task data includes the search type of the task to be processed and historical data of the processed tasks; The process of analyzing a search task includes: Based on historical data of processed tasks, perform characteristic analysis on different search types, and predict the load of tasks to be processed based on the results of the characteristic analysis, the search type of tasks to be processed, and the waiting time of tasks to be processed. The thread pool size adjustment strategy is determined based on system operation data and the predicted load of pending tasks. The process of performing feature analysis on different search types includes: Extract the CPU usage time series Oc(t1), Oc(t1), ..., Oc(tn) and memory usage time series Om(t1), Om(t1), ..., Om(tn) for each processed task from the historical data of processed tasks; A CPU usage curve was fitted based on the CPU usage time series of all processed tasks. ; Calculate the CPU usage time series and CPU usage curve for each processed task. Calculate the sum of squared residuals Sc; calculate the standard deviation of the sum of squared residuals Sc for all processed tasks. ; A memory usage curve was fitted based on the memory usage time series of all processed tasks. ; Calculate the memory usage time series and memory usage curve for each processed task. Calculate the residual sum of squares Sm; calculate the standard deviation of the residual sum of squares Sm for all processed tasks. ; By equation: Calculate and obtain reference CPU usage curves for various task search characteristics. and reference memory usage curve ; Obtain the reference CPU usage curve corresponding to the search type of the task to be processed. and reference memory usage curve ; in, Standard deviation The corresponding reference threshold, This is the first fixed coefficient; Standard deviation The corresponding reference threshold, This is the second fixed coefficient.

2. The high-performance, easily extensible unified search method based on Spring according to claim 1, characterized in that, The process of predicting the workload of tasks to be processed includes: Using the equations respectively: Calculate and obtain the predicted CPU load value and memory predicted load value Based on CPU predicted load value and memory predicted load value Predict the workload of tasks to be processed; Where m is the number of tasks to be processed, and j∈[1,m], This serves as a reference CPU usage curve for the characteristics of the j-th task to be processed. Indicates taking Maximum value This is a reference memory usage curve for the characteristics of the j-th task to be processed. Indicates taking Maximum value.

3. The high-performance, easily extensible unified search method based on Spring according to claim 2, characterized in that, The process of determining the thread pool size adjustment strategy includes: Through equations Calculate the required number of threads Q, and compare Q with the current number of threads Q0: If Q≥Q0, determine whether the system performance meets the requirements after increasing the number of threads: If the requirement is met, the thread pool size will be adjusted according to the required number of threads Q. If the conditions are not met, the thread pool size is determined based on system operating data and the difference between Q and Q0. If Q < Q0, determine whether to adjust the thread pool size based on system operating data and the difference between Q and Q0; in, This is a function that compares CPU predicted load value with the number of threads. A function to predict memory load value versus thread count. Indicates taking and The maximum value in.

4. The high-performance, easily extensible unified search method based on Spring according to claim 3, characterized in that, When Q≥Q0, the process of determining whether the system performance is satisfied after increasing the number of threads includes: Using the equations respectively: Calculate the remaining CPU percentage and remaining memory percentage CPU percentage and remaining memory percentage Compare with the corresponding thresholds respectively, when and If all values ​​are greater than the corresponding threshold, it is determined that the system performance can be satisfied after increasing the number of threads; otherwise, it is determined that the system performance cannot be satisfied after increasing the number of threads. in, This represents the current percentage of CPU idle time. Preset CPU percentage for single-threaded operation. This represents the current percentage of free memory. Preset memory allocation for single-threaded applications; When it is determined that increasing the number of threads will not meet system performance requirements, the process of determining the thread pool size includes: Through equations Calculate the thread pool size ; in, , Remaining CPU percentage and remaining memory percentage The corresponding threshold, The floor symbol is used for rounding down. Indicates taking and The minimum value in.

5. The high-performance, easily extensible unified search method based on Spring according to claim 4, characterized in that, When Q≥Q0, the process of determining whether to adjust the thread pool size based on system operating data and the difference between Q and Q0 includes: Through equations Calculate the thread pool size ; Where f(x) is a defined function, f(x) = x when x ≥ 0, otherwise f(x) = 0; , Remaining CPU percentage and remaining memory percentage The reference value, and satisfying > , > ; Indicates taking and The minimum value in.

6. The high-performance, easily extensible unified search method based on Spring according to claim 1, characterized in that, The search task is encapsulated as a CompletableFuture object, and asynchronous operation management is performed based on CompletableFuture in JAVA8.

Images