Message batch query method and device, equipment and storage medium

Abstract

The invention provides a message batch query method and device, equipment and a storage medium, and relates to the technical field of data processing.The method comprises the steps that firstly, request information is obtained, and the request information comprises an object storage file ID serving as a unique identifier of the request information; and then converting the request information into an asynchronous task, storing the asynchronous task in an asynchronous file consumption table, executing an asynchronous query task on the asynchronous file consumption table according to preset scheduling time, obtaining a query result through batch query operation, and writing the query result into an object storage file. By introducing an asynchronous processing and batch operation mechanism, efficient and ordered processing of a large number of query requests is realized. According to the method, the query processing capacity and the response speed of the system are remarkably improved, resource utilization is optimized, the system load is reduced, and therefore smoother and quicker query experience is provided for users.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a message batch query method, device, equipment and storage medium. Background Art

[0002] In today's digital age, enterprise informatization has become the key to driving business growth and improving operational efficiency. With the expansion of enterprise scale and the complexity of business, especially the rapid development of personal business middle office customer center business, data management and processing capabilities have become one of the important indicators to measure enterprise competitiveness. In these business scenarios, batch query of customer information is a common and critical demand, which is directly related to the enterprise's marketing strategy formulation, market analysis, customer service and other aspects.

[0003] However, traditional batch query functions often face many challenges in design and implementation. First, due to the different query requirements of different business departments (such as marketing centers, analysis centers, etc.), existing systems often need to be customized for each specific requirement. This not only increases development costs, but also makes the system structure complex and difficult to maintain and upgrade. Secondly, during batch queries, the processing of large amounts of data often places high demands on the server's memory and performance. Under traditional methods, problems such as memory overflow and low query efficiency are prone to occur. In addition, with the diversification of data sources (such as relational databases, NoSQL databases, cloud storage, etc.), how to achieve flexible switching and efficient query between multiple data sources has also become a problem that needs to be solved urgently. Summary of the invention

[0004] In view of the above problems, the present application provides a message batch query method, including the following contents:

[0005] In a first aspect, the present application provides a message batch query method, the method comprising:

[0006] Obtaining request information, wherein the request information includes an object storage file ID as a unique identifier of the request information;

[0007] Convert the request information into an asynchronous task and store it in an asynchronous file consumption table;

[0008] Execute an asynchronous query task on the asynchronous file consumption table according to a preset scheduling time;

[0009] The query results are obtained through batch query operations and written into the object storage file.

[0010] Optionally, before obtaining the query results through the batch query operation, the method further includes:

[0011] Check whether there are unprocessed tasks in the asynchronous file consumption table;

[0012] If it exists, get a pending task and lock the obtained pending task.

[0013] Optionally, the performing of the batch query operation includes:

[0014] Split the pending tasks into multiple small batches for query;

[0015] Write the query results to the object storage file;

[0016] After writing the query result to the object storage file, the large object data in the local memory is cleared.

[0017] Optionally, the executing the asynchronous query task includes:

[0018] Query customer information from multiple data sources;

[0019] The query results are spliced ​​according to the customer number range to ensure the integrity and consistency of the information.

[0020] Optionally, after writing the query result into the object storage file, the method further includes:

[0021] The ID of the object storage file is obtained, and the object storage file is sent to the caller through the message queue to complete the delivery of the query result.

[0022] Optionally, the batch query operation adopts a factory strategy mode, which supports flexible configuration and expansion of multiple data sources and query logics through configuration management and plug-in methods, thereby improving the scalability and adaptability of the system.

[0023] Optionally, the method further includes:

[0024] Monitor and optimize the execution of asynchronous tasks to ensure the timeliness and stability of task execution, and be able to promptly detect and handle delayed or failed tasks.

[0025] In a second aspect, the present application provides a message batch query device, the device comprising:

[0026] An acquiring unit, used for acquiring request information, wherein the request information includes an object storage file ID as a unique identifier of the request information;

[0027] A processing unit, used for converting the request information into an asynchronous task and storing the task in an asynchronous file consumption table;

[0028] A query unit, used to execute an asynchronous query task on the asynchronous file consumption table according to a preset scheduling time;

[0029] The storage unit is used to obtain query results through batch query operations and write the query results into an object storage file.

[0030] Optionally, the query unit is further used to query whether there are unprocessed tasks in the asynchronous file consumption table before obtaining the query result through the batch query operation;

[0031] If it exists, get a pending task and lock the obtained pending task.

[0032] Optionally, the query unit performs a batch query operation including:

[0033] Split the pending tasks into multiple small batches for query;

[0034] Write the query results to the object storage file;

[0035] After writing the query result to the object storage file, the large object data in the local memory is cleared.

[0036] Optionally, the query unit executes the asynchronous query task including:

[0037] Query customer information from multiple data sources;

[0038] The query results are spliced ​​according to the customer number range to ensure the integrity and consistency of the information.

[0039] Optionally, the device further comprises:

[0040] The transmission unit is used to obtain the ID of the object storage file after writing the query result into the object storage file, and send the object storage file to the caller through the message queue to complete the transmission of the query result.

[0041] Optionally, the query unit adopts a factory strategy mode for batch query operations, supporting flexible configuration and expansion of multiple data sources and query logics through configuration management and plug-in methods, thereby improving the scalability and adaptability of the system.

[0042] Optionally, the device further includes a monitoring unit for monitoring and optimizing the execution of asynchronous tasks to ensure the timeliness and stability of task execution and to be able to promptly detect and process delayed or failed tasks.

[0043] In a third aspect, the present application provides a device comprising a memory and a processor, wherein the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device executes the message batch query method introduced in any implementation of the first aspect.

[0044] In a fourth aspect, the present application provides a computer-readable storage medium, in which a code is stored. When the code is executed, the device executing the code implements the message batch query method introduced in any implementation of the first aspect.

[0045] The present application provides a message batch query method. When executing the method, first obtain the request information, the request information includes the object storage file ID as the unique identifier of the request information, then convert the request information into an asynchronous task and store it in the asynchronous file consumption table, execute the asynchronous query task on the asynchronous file consumption table according to the preset scheduling time, obtain the query result through the batch query operation, and write the query result into the object storage file. By introducing the asynchronous processing and batch operation mechanism, efficient and orderly processing of a large number of query requests is achieved. The method not only significantly improves the query processing capability and response speed of the system, but also optimizes resource utilization and reduces the system load, thereby providing users with a smoother and faster query experience. Specifically, by converting the request information into an asynchronous task and storing it in a special consumption table, the system can flexibly schedule resources and avoid the resource blocking problem caused by synchronous query. The preset scheduling time strategy further ensures that the query task can be executed at the right time, which not only avoids resource conflicts during peak hours, but also makes full use of idle resources during off-peak hours. The application of batch query operations greatly improves the efficiency of data retrieval and reduces the number of network transmissions and data processing time. At the same time, writing the query results into the object storage file in a centralized manner not only ensures the consistency and integrity of the data, but also makes it convenient for users to obtain all results at one time, reduces waiting time, and improves user satisfaction. In addition, the method also simplifies the system architecture and business logic, making the system easier to expand and maintain. As the volume of business grows, the system can easily cope with it by adding asynchronous processing units and storage resources without the need for large-scale modifications to the existing architecture. In summary, the message batch query method provided in this application, with its high efficiency, flexibility, and reliability, has brought significant overall beneficial effects in improving system performance, optimizing user experience, and reducing operation and maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A flowchart of a message batch query method provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of the structure of a message batch query device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] When performing the message batch query method in the embodiment of the present application, multiple components such as the function caller analysis center, object storage, function provider customer center, MQ message middle platform, and xxljob work together to achieve an efficient and reliable message batch query function. The following is a detailed analysis of the role of these components in the message batch query process:

[0051] 1. Function Caller Analysis Center

[0052] Function: Demand reception: Receive batch query requests from external systems or users, and parse the parameters and conditions in the requests.

[0053] Task scheduling: According to the query request, schedule the corresponding query task to the subsequent processing component.

[0054] Result aggregation: Collect the query results returned by each processing component, aggregate and organize them, and finally return them to the caller.

[0055] 2. Object storage platform

[0056] Function: Data storage: used to store intermediate results and final query results generated during the query process. These results are stored in the object storage system in the form of files or objects for subsequent access and processing.

[0057] Data persistence: ensures that query results will not be lost due to system restart or failure, providing data security and reliability.

[0058] 3. Function Provider Customer Center

[0059] Function: Data source provision: As a data provider, it stores and manages the data that needs to be queried. This data may come from multiple different data sources, such as databases, file systems, etc.

[0060] Data interface: Provides a data access interface for query tasks to call to obtain the required data.

[0061] 4.MQ message center

[0062] Function: Message transmission: During the query process, the MQ message center is responsible for transmitting messages such as query tasks and query results between different components. It implements asynchronous communication and improves the system's response speed and throughput.

[0063] Load balancing: Through the message queue mechanism, the MQ message center can automatically achieve load balancing, evenly distribute query tasks to multiple processing nodes, and improve the system's parallel processing capabilities.

[0064] 5.xxljob

[0065] Function: Task scheduling and execution: xxljob is a distributed task scheduling platform, which is responsible for receiving query tasks from the function caller analysis center, and scheduling and executing them according to the task configuration. xxljob supports multiple task types such as scheduled tasks and one-time tasks to meet different query requirements.

[0066] Task monitoring: xxljob also provides a task monitoring function, which can monitor the execution status, execution results and other information of the task in real time to ensure the smooth execution of the query task.

[0067] The message batch query method in this application is introduced below in conjunction with specific embodiments. Figure 1 A flowchart of a message batch query method provided in an embodiment of the present application. Figure 1 As shown, the message batch query method provided in the embodiment of the present application may include:

[0068] S101. Obtain request information, where the request information includes an object storage file ID as a unique identifier of the request information.

[0069] Request information refers to the information packet sent by the client (such as user equipment, application, etc.) to the server to request a certain service or data. This information usually contains all the necessary data required to perform a specific operation.

[0070] The system first receives the request information from the client. These request information contain the key data necessary to perform batch query operations, especially the object storage file ID, which is used as the unique identifier of the request information to locate and query the corresponding data in the database. The object storage file ID is usually a unique identifier used to uniquely identify a file or object in the object storage system.

[0071] In the embodiment of the present application, the function caller analysis center initiates a data query, the object storage platform analysis center writes the query conditions into a file and uploads it to the object storage, and obtains the object storage file id. The MQ message center analysis center sends the object storage file id to the customer center via mq, the topic is 000001%TOPIC_ICMC_ASYNC_TASK, and the event id is the custom id assigned to the scenario.

[0072] S102: Convert the request information into an asynchronous task and store it in an asynchronous file consumption table.

[0073] Asynchronous task: refers to a task that is not executed immediately but is scheduled by the system to be executed at a certain point in the future. Asynchronous tasks allow the program to continue to perform other operations while waiting for the task to complete, thereby improving the concurrency and responsiveness of the program.

[0074] Asynchronous file consumption table: A data table used to store asynchronous tasks to be processed. These tasks are usually stored in a table in a certain order (such as priority, arrival time, etc.), waiting for the system to schedule and execute according to preset rules.

[0075] In this step, the system converts the received request information into an asynchronous task and stores it in the asynchronous file consumption table. The purpose of this step is to convert the synchronous query request into an asynchronous processing mode so that the system can more flexibly schedule resources and process multiple query requests. By storing the request information in the asynchronous file consumption table, the system can arrange the execution order of tasks according to the preset scheduling strategy (such as time, priority, etc.), thereby optimizing resource utilization and improving query efficiency.

[0076] In the embodiment of the present application, after receiving the information, the customer center consumes the information and inserts an asynchronous task into the asynchronous file consumption table CMC.ASYNC_TASK.

[0077] S103: Execute an asynchronous query task on the asynchronous file consumption table according to a preset scheduling time.

[0078] Preset scheduling time: refers to the time point or time interval set in advance by the system to trigger the execution of asynchronous tasks. These time settings are usually based on factors such as business requirements, system load, and resource availability.

[0079] In step S103, the system executes the tasks in the asynchronous file consumption table according to the preset scheduling time. This means that the system will check the asynchronous file consumption table according to the set time rules (such as executing once every hour, executing every morning, etc.) and schedule the tasks in the table for execution. This scheduling method helps the system balance the load and avoid resource overload caused by executing a large number of query tasks during peak hours. At the same time, by presetting the scheduling time, the system can also ensure that the query tasks are executed at the right time to meet business needs and data timeliness requirements.

[0080] In an embodiment of the present application, the client center asynchronous file consumption table is periodically scheduled through the timer asyncTask, for example, once every 20 seconds.

[0081] S104: Obtain query results through batch query operations, and write the query results into an object storage file.

[0082] Batch query operation: refers to the process of performing query operations on multiple data items at one time. Compared with single query, batch query can significantly improve query efficiency and reduce network transmission times and data processing time.

[0083] Object storage file: A storage system for storing unstructured data. It allows users to store data in the form of objects. Each object contains the data itself, metadata (such as file name, size, creation time, etc.) and a unique identifier (such as object storage file ID).

[0084] The system retrieves data matching the requested information from the database through batch query operations. These query operations may be based on the object storage file ID or other related conditions. Once the query is completed, the system will organize the query results and write them to the specified object storage file. The advantage of this is that users can get all the results at once after the query is completed without waiting for the response of each individual query. At the same time, storing the query results in the object storage file also helps with the long-term preservation and sharing of data. In addition, object storage systems are usually highly reliable and scalable, and can meet the needs of large-scale data storage and access.

[0085] In an implementation of the embodiment of the present application, before obtaining the query results through the batch query operation, the method also includes: querying whether there are unprocessed tasks in the asynchronous file consumption table; if so, obtaining a pending task and locking the obtained pending task.

[0086] Before executing a batch query operation, the system will first check whether there are unprocessed tasks in the asynchronous file consumption table. If so, a pending task is obtained and locked to prevent multiple execution units from processing the same task at the same time, ensuring data consistency and task atomicity. This prevents duplicate task processing and ensures data integrity and consistency during query and processing.

[0087] In one implementation of the embodiment of the present application, the performing of the batch query operation includes:

[0088] Split the task to be processed into multiple small batches for query; write the query results into an object storage file; and clear the large object data in the local memory after writing the query results into the object storage file.

[0089] To-be-processed tasks: refer to query tasks that need to be executed. These tasks usually contain information such as specific query conditions and target data sources.

[0090] Mini-batch: In order to process large-scale data or improve query efficiency, the task to be processed is divided into several smaller query units, each of which is called a mini-batch. The size of the mini-batch can be adjusted according to the actual situation to achieve the best query performance.

[0091] Object storage file: A file system used to store unstructured data. It allows users to store data in the form of objects. Each object contains the data itself, metadata (such as file name, size, creation time, etc.) and a unique identifier. In this scenario, object storage files are used to store batch query results.

[0092] Large object data in local memory: During query execution, the system may store query results or intermediate data in local memory for quick access and processing. However, for large-scale data queries, this data may occupy a large amount of memory space, leading to tight memory resources. Therefore, after writing query results to object storage files, these large object data need to be cleared in time to free up memory resources.

[0093] Improve query efficiency and system stability. By splitting the pending tasks into multiple small batches for querying, the system load can be dispersed to avoid excessive pressure on system resources caused by a single query operation. At the same time, small batch queries can also reduce the risk of query failure, because even if a small batch query fails, it will not affect the execution of the entire task. Split the pending tasks into multiple small batches according to the preset batch size or conditions (such as data volume, time, etc.). Then, perform independent query operations on each small batch.

[0094] Object storage files provide a safe and reliable storage mechanism to ensure that query results will not be lost due to system restart or failure. At the same time, storing query results in object storage files also facilitates subsequent data analysis and processing. After the query operation is completed, the system will organize the query results and write them to the specified object storage file. This process may include operations such as serialization, compression, and encryption of data to ensure data security and accessibility.

[0095] Clearing the large object data in the local memory after writing the query results to the object storage file can release memory resources and improve system performance. During the query execution process, the system may store a large amount of data in the local memory. If these data are not cleaned up in time, it may lead to tight memory resources and affect the normal operation of the system. Therefore, after writing the query results to the object storage file, the system needs to clear the large object data in the local memory to release memory resources for subsequent operations. The system can clear the large object data in the local memory by calling the memory management interface or the garbage collection mechanism. These operations are usually automatically executed after the query results are written to the object storage file to ensure the timely release of memory resources.

[0096] In one implementation of the embodiment of the present application, the executing asynchronous query task includes: querying customer information from multiple data sources; and splicing the query results according to the customer number range to ensure the integrity and consistency of the information.

[0097] Querying customer information from multiple data sources is for data source diversity. Databases may include: ES, Oracle, PHOENIX, ECIF, etc. Customer number is usually the number used to uniquely identify customers within the enterprise. When splicing query results from different data sources, sorting and splicing by customer number range can ensure the logical order and integrity of the data. For example, all customer information with customer numbers between 1000-1999 can be spliced ​​together, followed by customer information with numbers between 2000-2999, and so on. Since customer information may come from different data sources, and the data update frequency and synchronization mechanism of these data sources may be different, special attention should be paid to data consistency during the splicing process. The system may need to implement some data cleaning and deduplication mechanisms to ensure that the customer information in the final splicing result is accurate, unique and up-to-date.

[0098] In one implementation of the embodiment of the present application, after writing the query result into the object storage file, the method further includes: obtaining the ID of the object storage file, sending the object storage file to the caller through a message queue, and completing the delivery of the query result.

[0099] The object storage system assigns a unique identifier (ID) to each stored file. This ID is globally unique and can be used to retrieve and access files stored in the object storage. Therefore, after writing the query results to the object storage file, the system needs to obtain the ID of this file so that it can be used to reference and transfer files later.

[0100] Send the object storage file to the caller through the message queue: After obtaining the ID of the object storage file, the system will encapsulate this ID (or other necessary information including the ID, such as the file's access URL, permission token, etc.) into a message and send it to the caller through the message queue. As an asynchronous communication mechanism, the message queue can ensure that messages are reliably delivered between the sender and the receiver, even if they are not directly connected physically or in time. Through the message queue, the caller can pull messages from the queue at the appropriate time and access the object storage file based on the information in the message to obtain the query results.

[0101] In one implementation of the embodiment of the present application, the batch query operation adopts a factory strategy mode, which supports flexible configuration and expansion of multiple data sources and query logic through configuration management, plug-in, etc., thereby improving the scalability and adaptability of the system.

[0102] Through the factory strategy pattern, this design enables the system to flexibly process information from different data sources, and can select the appropriate data source for operation according to needs, enhancing the scalability and adaptability of the system. In order to avoid the need to query multiple tables or multiple data sources for data splicing in some situations, the primary and secondary data sources are supported here, and the secondary data sources are used to supplement the query results of the primary table.

[0103] In one implementation of the embodiment of the present application, the method further includes: monitoring and optimizing the execution of asynchronous tasks to ensure the timeliness and stability of task execution, and being able to promptly discover and process delayed or failed tasks.

[0104] The purpose of monitoring is to track the execution status of asynchronous tasks in real time or periodically, including the task start time, execution time, end time, execution result (success, failure, delay, etc.) and any related performance indicators (such as CPU usage, memory usage, network latency, etc.).

[0105] This can be achieved through the following methods, such as logging: recording detailed log information at each stage of task execution for subsequent analysis and auditing. Monitoring tools: using specialized monitoring tools (such as Prometheus, Grafana, ELK Stack, etc.) to collect, process and display monitoring data. Alert system: setting alert thresholds. When the monitoring data exceeds these thresholds (such as task execution time is too long, failure rate is too high, etc.), the system automatically triggers an alarm and notifies relevant personnel.

[0106] The purpose of optimization is to improve the efficiency, stability and predictability of asynchronous task execution and reduce delays and failures. This can be achieved through the following methods. For example, task scheduling optimization: according to the nature of the task and the usage of resources, the execution order and parallelism of tasks can be reasonably scheduled. For example, for IO-intensive tasks, the parallelism can be increased to make full use of hardware resources; for CPU-intensive tasks, the parallelism needs to be carefully controlled to avoid resource contention.

[0107] Resource allocation optimization: Dynamically adjust resource allocation (such as CPU time slices, memory size, network bandwidth, etc.) according to task requirements and system resource conditions. This ensures that critical tasks can obtain sufficient resources to be completed on time while avoiding resource waste.

[0108] Error handling: Design a complete error handling mechanism for asynchronous tasks, including retry strategies, error logging and reporting, etc. When a task fails, the system can automatically retry or take other remedial measures to reduce the impact of task failure on the overall system.

[0109] Performance tuning: Regularly analyze monitoring data, identify bottlenecks and hot spots, and optimize code, database, network, etc. For example, you can optimize database query statements, adjust JVM parameters, increase cache, etc. to improve task execution efficiency.

[0110] In the above embodiment of the present application, by converting the request information into an asynchronous task and storing it in the asynchronous file consumption table, the system resources are effectively utilized, and the system congestion and performance degradation caused by the simultaneous processing of a large number of requests in the traditional synchronous query mode are avoided. At the same time, the preset scheduling time mechanism enables the system to execute the query task in an orderly and efficient manner, further improving the query efficiency. The read-while-write processing method is adopted, and the results are written to the object storage file in real time during the query process, which avoids a large amount of data from being resident in the memory for a long time, thereby significantly reducing the memory consumption of the service. This is particularly important for processing scenarios with large-scale data sets, and helps to improve the stability and reliability of the system. By decoupling the query request from the specific query logic and introducing the factory strategy mode to support multi-data source queries, the method enables the system to easily cope with the diverse query needs of different business departments. At the same time, the modular design idea makes the development of new query scenarios simpler and faster, reduces the development cost and time cycle, and enhances the reusability and flexibility of the system. Due to the improvement of query efficiency and the reduction of memory consumption, users can obtain query results faster, thereby improving the user experience. In addition, through asynchronous processing, users do not need to wait for query results for a long time, and can carry out subsequent work more efficiently. Support for large data processing: This method is particularly suitable for batch query tasks that process large amounts of data. By using object storage files as a medium for data exchange, the system can efficiently process massive amounts of data and meet the growing data processing needs of enterprises.

[0111] To sum up, the message batch query method provided in this application shows significant beneficial effects in improving query efficiency, reducing memory consumption, enhancing system reusability and flexibility, optimizing user experience, and supporting large data volume processing, which is of great significance for improving the level of enterprise informatization and promoting business growth.

[0112] The above are some specific implementations of a message batch query method provided in the embodiment of the present application. Based on this, the present application also provides a corresponding device. The device provided in the embodiment of the present application will be introduced from the perspective of functional modularization.

[0113] Figure 2 A schematic diagram of the structure of a message batch query device provided in an embodiment of the present application. Figure 2 As shown, the message batch query device 200 provided in the embodiment of the present application includes:

[0114] An acquiring unit 210 is used to acquire request information, wherein the request information includes an object storage file ID as a unique identifier of the request information;

[0115] The processing unit 220 is used to convert the request information into an asynchronous task and store it in an asynchronous file consumption table;

[0116] A query unit 230, configured to execute an asynchronous query task on the asynchronous file consumption table according to a preset scheduling time;

[0117] The storage unit 240 is used to obtain query results through batch query operations and write the query results into an object storage file.

[0118] In an implementation of the embodiment of the present application, the query unit is further used to query whether there are unprocessed tasks in the asynchronous file consumption table before obtaining the query result through the batch query operation;

[0119] If it exists, get a pending task and lock the obtained pending task.

[0120] In an implementation of the embodiment of the present application, the query unit performs a batch query operation including:

[0121] Split the pending tasks into multiple small batches for query;

[0122] Write the query results to the object storage file;

[0123] After writing the query result to the object storage file, the large object data in the local memory is cleared.

[0124] In an implementation of the embodiment of the present application, the query unit executes the asynchronous query task including:

[0125] Query customer information from multiple data sources;

[0126] The query results are spliced ​​according to the customer number range to ensure the integrity and consistency of the information.

[0127] In one implementation of the embodiment of the present application, the device further includes:

[0128] The transmission unit is used to obtain the ID of the object storage file after writing the query result into the object storage file, and send the object storage file to the caller through the message queue to complete the transmission of the query result.

[0129] In one implementation of the embodiment of the present application, the query unit adopts a factory strategy mode for batch query operations, supporting flexible configuration and expansion of multiple data sources and query logic through configuration management, plug-in, etc., thereby improving the scalability and adaptability of the system.

[0130] In one implementation of the embodiment of the present application, the device also includes a monitoring unit for monitoring and optimizing the execution of asynchronous tasks to ensure the timeliness and stability of task execution and to be able to promptly detect and handle delayed or failed tasks.

[0131] The embodiments of the present application also provide corresponding devices and computer storage media for implementing the solutions provided by the embodiments of the present application.

[0132] The device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device executes the method described in any embodiment of the present application.

[0133] The computer storage medium stores codes, and when the codes are executed, a device executing the codes implements the method described in any embodiment of the present application.

[0134] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0135] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0136] It should also be noted that the various embodiments in this specification are described in a progressive manner, and the same and similar parts between the various embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments. The device and apparatus embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components indicated as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0137] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method, characterized in that The method comprises: Obtaining request information, wherein the request information includes an object storage file ID as a unique identifier of the request information; Convert the request information into an asynchronous task and store it in an asynchronous file consumption table; Execute an asynchronous query task on the asynchronous file consumption table according to a preset scheduling time; The query results are obtained through batch query operations and written into the object storage file.

2. The method according to claim 1, characterized in that Before obtaining the query results through the batch query operation, the method further includes: Check whether there are unprocessed tasks in the asynchronous file consumption table; If it exists, get a pending task and lock the obtained pending task.

3. The method according to claim 1, characterized in that The batch query operation includes: Split the pending tasks into multiple small batches for query; Write the query results to the object storage file; After writing the query result to the object storage file, the large object data in the local memory is cleared.

4. The method according to claim 1, characterized in that The asynchronous query task execution includes: Query customer information from multiple data sources; The query results are spliced ​​according to the customer number range to ensure the integrity and consistency of the information.

5. The method according to claim 1, characterized in that After writing the query result into the object storage file, the method further includes: The ID of the object storage file is obtained, and the object storage file is sent to the caller through the message queue to complete the delivery of the query result.

6. The method according to any one of claims 1 to 5, characterized in that The batch query operation adopts a factory strategy mode, which supports flexible configuration and expansion of multiple data sources and query logics through configuration management and plug-in methods, thereby improving the scalability and adaptability of the system.

7. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Monitor and optimize the execution of asynchronous tasks to ensure the timeliness and stability of task execution, and be able to promptly detect and handle delayed or failed tasks.

8. A message batch query device, characterized in that: The system includes: An acquiring unit, used for acquiring request information, wherein the request information includes an object storage file ID as a unique identifier of the request information; A processing unit, used for converting the request information into an asynchronous task and storing the task in an asynchronous file consumption table; A query unit, used to execute an asynchronous query task on the asynchronous file consumption table according to a preset scheduling time; The storage unit is used to obtain query results through batch query operations and write the query results into an object storage file.

9. A computing device, characterized in that The computing device includes: a memory and a processor; The memory is used to store computer programs; The processor is configured to implement the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images