Data acquisition method and device, electronic equipment and storage medium
By automatically determining and executing the query operation of data acquisition requests, based on the filtering dimension and target data table, the problem of low data acquisition efficiency in the existing technology is solved, and efficient data acquisition and resource management are achieved.
Patent Information
- Application Number
- CN202410954435.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-08
AI Technical Summary
During the operation and maintenance of business systems, for operation and maintenance work orders of different demand parties, the existing technology needs to customize the writing of SQL scripts to obtain data, resulting in a prolonged development cycle, low data acquisition efficiency, and data extraction occurs during concurrent execution. Slow speed problem.
Propose a data acquisition method, which can determine the evaluation strategy and target data table based on the filtering dimension, automatically determine the data acquisition time or amount, and perform query operations at the target time to obtain target data from the target data table, saving I went through the process of writing scripts manually to avoid resource competition.
It improves the efficiency of data acquisition, reduces the development cycle, and improves the speed of data acquisition, especially effectively avoids resource competition during concurrent execution, and improves system performance.
Smart Images

Figure CN118916390A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data acquisition method, device, electronic device and storage medium. Background Art
[0002] In the daily operation and maintenance of business systems, it is often necessary to customize SQL scripts and execute them on the database side to obtain data for operation and maintenance work orders from different demand parties. This method prolongs the development cycle and reduces the efficiency of data acquisition. At the same time, when executing a large number of data acquisition operations concurrently, the system often encounters the problem of slow data extraction speed, which further affects the efficiency of data acquisition.
[0003] Therefore, there is an urgent need for an efficient data acquisition method. Summary of the invention
[0004] The present application aims to solve one of the technical problems in the related art at least to some extent.
[0005] To this end, the first objective of the present application is to propose a data acquisition method to improve the efficiency of data acquisition.
[0006] The second objective of the present application is to provide a data acquisition device.
[0007] The third objective of the present application is to provide an electronic device.
[0008] A fourth objective of the present application is to provide a computer-readable storage medium.
[0009] A fifth object of the present application is to provide a computer program product.
[0010] To achieve the above-mentioned purpose, the first embodiment of the present application proposes a data acquisition method, including:
[0011] In response to receiving the data acquisition request, determining an evaluation strategy and a target data table based on a screening dimension in the data acquisition request;
[0012] Based on the evaluation strategy, determine the duration or amount of data acquisition;
[0013] Determine the target time for executing the query operation corresponding to the data acquisition request based on the data acquisition duration or data acquisition amount;
[0014] Execute a query operation at the target time to obtain the target data from the target data table.
[0015] To achieve the above-mentioned purpose, a second aspect of the present application provides a data acquisition device, including:
[0016] A matching module, configured to determine an evaluation strategy and a target data table based on a screening dimension in the data acquisition request in response to receiving the data acquisition request;
[0017] A first determination module is used to determine the data acquisition time or data acquisition amount based on the evaluation strategy;
[0018] A second determination module is used to determine a target time for executing a query operation corresponding to a data acquisition request based on a data acquisition duration or a data acquisition amount;
[0019] The execution module is used to execute the query operation at the target time to obtain the target data from the target data table.
[0020] To achieve the above-mentioned purpose, the third aspect of the present application provides an electronic device, including:
[0021] at least one processor; and
[0022] a memory communicatively connected to at least one processor; wherein,
[0023] The memory stores instructions that can be executed by at least one processor. The instructions are executed by the at least one processor so that the at least one processor can execute the method of the above embodiment.
[0024] To achieve the above objectives, the fourth aspect of the present application proposes a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method according to the above embodiment.
[0025] To achieve the above objectives, the fifth aspect of the present application proposes a computer program product, including a computer program, which implements the method of the above embodiment when executed by a processor.
[0026] The data acquisition method, device, electronic device and storage medium provided by the present application, in response to receiving a data acquisition request, determine the evaluation strategy and target data table based on the screening dimension in the data acquisition request, and then determine the data acquisition duration or data acquisition amount based on the evaluation strategy, so as to determine the target time for executing the query operation corresponding to the data acquisition request based on the data acquisition duration or data acquisition amount, and then execute the query operation at the target time to obtain the target data from the target data table. Thus, by automatically determining and executing the query operation corresponding to the data acquisition request, the process of manually writing the data acquisition script is omitted. At the same time, according to the data acquisition duration or data acquisition amount, the timing (target time) of executing the query operation is determined to effectively avoid resource competition. Thereby improving the efficiency of data acquisition.
[0027] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0029] Figure 1 A flowchart of a data acquisition method provided in an embodiment of the present application;
[0030] Figure 2 A flowchart of another data acquisition method provided in an embodiment of the present application;
[0031] Figure 3 A flowchart of another data acquisition method provided in an embodiment of the present application;
[0032] Figure 4 A flowchart of another data acquisition method provided in an embodiment of the present application;
[0033] Figure 5 A schematic diagram of the structure of a data acquisition device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0034] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0035] The data acquisition method and device according to the embodiments of the present application are described below with reference to the accompanying drawings.
[0036] The data acquisition method of the embodiment of the present application is executed by the data acquisition device provided in the embodiment of the present application. The device can be configured in a computer device or a terminal device to improve the efficiency of data acquisition.
[0037] Figure 1 A flowchart of a data acquisition method provided in an embodiment of the present application.
[0038] like Figure 1 As shown, the data acquisition method includes the following steps:
[0039] Step 101, in response to receiving a data acquisition request, determining an evaluation strategy and a target data table based on a screening dimension in the data acquisition request.
[0040] Among them, the evaluation strategy can be used to determine the duration of data acquisition, or to determine the amount of data acquisition, etc., and this application does not impose any restrictions on this.
[0041] Different business data is stored in different data tables, and different business needs require different data. Therefore, after receiving a data acquisition request, it is necessary to determine the filtering dimensions corresponding to the data to be acquired and the target data table where it is stored. This can achieve automatic data extraction and improve data acquisition efficiency. Among them, the filtering dimensions correspond to the fields in the target data table.
[0042] In this application, the supported configuration filtering dimensions and the input boxes corresponding to each filtering dimension can be set in the client. After that, after the user enters the filtering condition content in the input box corresponding to each filtering dimension in the client, data acquisition can be triggered. As a result, the client can generate a data acquisition request based on each filtering dimension and its corresponding filtering condition content (i.e., feature value), and send the data acquisition request to the data acquisition device. The data acquisition device can determine the target data table corresponding to the filtering dimension by querying the data table mapping information. Among them, the data table mapping information contains the association relationship between the filtering dimension and the data table. In addition, the client also supports setting business information, and sending the business information to the data acquisition device.
[0043] Optionally, the user can also upload a data acquisition request work order to the client and trigger data acquisition. As a result, the client can generate a data acquisition request based on the data acquisition request work order and send the data acquisition request to the data acquisition device. After that, the data acquisition device can parse the data acquisition request work order to determine the filtering dimensions and the characteristic values corresponding to each filtering dimension. In addition, the data acquisition request work order can also include business information. After parsing the data acquisition work order to obtain business information, the target data table corresponding to the business information and the filtering dimension can be determined by querying the data table mapping information. Among them, the data table mapping information contains the relationship between business information, filtering dimensions and data tables.
[0044] In this application, different types of screening dimensions correspond to different evaluation strategies. The evaluation strategy corresponding to the screening dimension can be pre-set in the system. Then, the evaluation strategy corresponding to the screening dimension can be determined by query.
[0045] Step 102: Determine the data acquisition duration or data acquisition amount based on the evaluation strategy.
[0046] In this application, the specific contents of the evaluation strategy are as follows:
[0047] Method 1: When the screening dimension is a continuous type, such as the screening dimension is the insurance time, total premium, etc., the first range corresponding to all the data under the screening dimension in the target data table and the second range corresponding to the characteristic value under the screening dimension can be determined. Then, the proportion of the second range to the first range is determined. Then, the ratio is multiplied by the total number of data stored in the target data table to determine the amount of data to be acquired.
[0048] Method 2: When the screening dimension is a discrete type, such as the screening dimension is an identification bit (gender, etc.), a status bit (whether it is valid, etc.), etc., the preset ratio corresponding to the characteristic value under the screening dimension can be determined. Then, the ratio is multiplied by the total number of data stored in the target data table to determine the amount of data acquisition. Among them, the preset ratio can be determined in advance by statistically analyzing the data stored in the target data table at historical moments. For example, at the end of last year, the corresponding ratio of adults in a product purchase record table was 80%, and the corresponding ratio of minors was 20%. Then, the preset ratio corresponding to adults can be pre-set in the system to 80%, and the preset ratio corresponding to minors can be pre-set to 20%.
[0049] Method three: When there are multiple filtering dimensions and all of them correspond to the same target data table, you can use the above methods one and two to determine the data acquisition amount corresponding to each filtering dimension, and then determine the final data acquisition amount as the average of the sum of the data acquisition amount corresponding to each filtering dimension and the maximum value of the data acquisition amount corresponding to each filtering dimension.
[0050] Method 4: When there are multiple screening dimensions and the multiple screening dimensions correspond to different target data tables, the query operation may be an associated query. At this time, when the total amount of data in the target data table is greater than the preset threshold, it is determined that the data acquisition time is greater than the first preset threshold. When the total amount of data in the target data table is less than or equal to the preset threshold, it is determined that the data acquisition time is less than the first preset threshold.
[0051] Step 103: Determine a target time for executing a query operation corresponding to the data acquisition request based on the data acquisition duration or the data acquisition amount.
[0052] In the present application, when the data acquisition duration is greater than or equal to the first preset threshold, the time when the number of database operations executed simultaneously is less than the second preset threshold is determined as the target time. In addition, when the data acquisition amount is greater than or equal to the third preset threshold, the time when the number of database operations executed simultaneously is less than the second preset threshold is determined as the target time. This can effectively avoid long-term occupation of resources, avoid resource competition, and avoid affecting production business.
[0053] The moment when the number of database operations executed simultaneously is less than the second preset threshold can be determined by statistical analysis of database operations executed at historical moments, or by the number of database operations executed at the next moment.
[0054] For example, by statistically analyzing the database operations executed at historical moments, it is determined that the number of database operations executed at each moment between 12:00 p.m. and 6:00 a.m. is less than the second preset threshold, and any moment between 12:00 p.m. and 6:00 a.m. can be determined as the target time. Alternatively, the number of executed database operations is continuously monitored, and when the number of concurrently executed database operations at a certain moment in the future is less than the second preset threshold, the moment is determined as the target time.
[0055] When the data acquisition duration is less than the first preset threshold, or the data acquisition amount is less than the third preset threshold, the query operation corresponding to the data acquisition request can be executed immediately, thereby improving the efficiency of data acquisition.
[0056] Step 104, performing a query operation at a target time to obtain target data from a target data table.
[0057] In this application, a template script for query operations corresponding to common data acquisition request template work orders can be pre-set in the system. Afterwards, the acquired data acquisition request work order can be semantically matched with the data acquisition request template work order to determine the script corresponding to the data acquisition request. Then, the screening dimension and its corresponding feature value can be combined with the script corresponding to the data acquisition request to generate a target script. The target script can be executed at the target time to obtain the target data from the target data table.
[0058] Optionally, in order to ensure data security, the target user's authority in the data acquisition request may be reviewed, and if the target user's authority in the data acquisition request is reviewed and approved, the target data is fed back to the target user. If the target user's authority is not reviewed and approved, the data acquisition request is determined to have failed, and the target data is not fed back to the target user.
[0059] The method of reviewing the authority of the target user in the data acquisition request can be: sending a review request to a review user with review authority, and after receiving the review user's feedback information on the review request, sending the target data to the target user. Alternatively, the data acquisition range corresponding to different roles is pre-set in the system, and then, when the data acquisition range corresponding to the role to which the target user belongs contains the target data, the target data is fed back to the target user.
[0060] Optionally, when the target data contains sensitive data, mask processing is performed on the sensitive data to avoid leakage of the sensitive data.
[0061] In this application, in response to receiving a data acquisition request, an evaluation strategy and a target data table are determined based on the screening dimension in the data acquisition request. After that, based on the evaluation strategy, the data acquisition duration or the data acquisition amount is determined, and based on the data acquisition duration or the data acquisition amount, the target time for executing the query operation corresponding to the data acquisition request is determined, and then the query operation is executed at the target time to obtain the target data from the target data table. Thus, by automatically determining and executing the query operation corresponding to the data acquisition request, the process of manually writing the data acquisition script is omitted. At the same time, according to the data acquisition duration or the data acquisition amount, the timing (target time) of executing the query operation is determined to effectively avoid resource competition. Thereby improving the efficiency of data acquisition.
[0062] Figure 2 A flowchart of a data acquisition method provided in an embodiment of the present application.
[0063] like Figure 2 As shown, the data acquisition method includes the following steps:
[0064] Step 201, when the current moment is within a preset time period, in response to receiving a data acquisition request, determine an evaluation strategy and a target data table based on the screening dimensions in the data acquisition request, wherein the preset time period is a time period in which the total amount of historical data acquisition requests received within a historical time period is greater than a preset threshold.
[0065] According to business demand experience, the demand for data acquisition has a certain time pattern. For example, there are usually more data acquisition needs at the beginning and end of the month. Therefore, the periodic law of data acquisition can be identified and confirmed by counting the frequency of historical data acquisition requests within the historical time period. During the identified peak period, that is, the time period with intensive historical data acquisition requests, the data acquisition method of any embodiment of the present application is adopted to intelligently schedule resources and reduce the system burden to ensure efficient data acquisition. During the off-peak period, that is, the time period with fewer historical data acquisition requests, when a data acquisition request is received, the query operation corresponding to the data acquisition request can be directly determined and executed to obtain the target data. This can not only flexibly respond to demand fluctuations, but also ensure the efficient implementation of the overall data acquisition process, improve system performance and response speed, and thus improve the efficiency of data acquisition.
[0066] For example, the historical time period may be each month in the first six months. When statistics determine that the total number of historical data acquisition requests received within the first five days of the month and the last five days of the month in the first six months is greater than the preset threshold, the first five days of the month and the last five days of the month are determined as the preset time period. Afterwards, when a data acquisition request is received within the first five days of the month and the last five days of the month, the evaluation strategy is matched based on the screening dimension in the data acquisition request to determine the target time for executing the query operation corresponding to the data acquisition request based on the evaluation strategy, and acquire the target data.
[0067] Step 202: Determine the data acquisition duration or data acquisition amount based on the evaluation strategy.
[0068] Step 203: Determine a target time for executing a query operation corresponding to the data acquisition request based on the data acquisition duration or the data acquisition amount.
[0069] Step 204, performing a query operation at a target time to obtain target data from a target data table.
[0070] In the present application, the specific implementation process of steps 202 to 204 can be found in the detailed description of any embodiment of the present application and will not be repeated here.
[0071] In this application, when the current moment is within the preset time period, in response to receiving a data acquisition request, an evaluation strategy and a target database are determined based on the screening dimension in the data acquisition request, wherein the preset time period is a time period in which the total amount of historical data acquisition requests received within the historical time period is greater than a preset threshold, and then, based on the evaluation strategy, the data acquisition duration or data acquisition amount is determined, and based on the data acquisition duration or data acquisition amount, the target time for executing the query operation corresponding to the data acquisition request is determined, and then the query operation is executed at the target time to obtain the target data from the target database. Therefore, by analyzing the total amount of historical data acquisition requests and determining whether to execute the timing of the data acquisition method of this application, it is possible to flexibly respond to demand fluctuations and ensure the efficient execution of the overall data acquisition process, thereby improving system performance and response speed, and thus improving the efficiency of data acquisition.
[0072] Figure 3 A flowchart of a data acquisition method provided in an embodiment of the present application.
[0073] like Figure 3 As shown, the data acquisition method includes the following steps:
[0074] Step 301, in response to receiving a data acquisition request, determining an evaluation strategy and a target data table based on a screening dimension in the data acquisition request.
[0075] Step 302: Determine the data acquisition duration or data acquisition amount based on the evaluation strategy.
[0076] Step 303: Determine a target time for executing a query operation corresponding to the data acquisition request based on the data acquisition duration or the data acquisition amount.
[0077] In the present application, the specific implementation process of steps 301 to 303 can be found in the detailed description of any embodiment of the present application and will not be repeated here.
[0078] Step 304, while executing the query operation at the target time, monitor the system performance and obtain performance data, wherein the performance data includes CPU usage, memory usage, IO usage, and the number of database connections.
[0079] In the present application, during the process of executing the query operation at the target time, the performance of the current data acquisition system can be monitored to obtain performance data.
[0080] Step 305: When the performance data is abnormal, execute the abnormality control program.
[0081] In the present application, it is possible to determine whether the performance data is abnormal, so that in the event that the performance data is abnormal, an abnormality control program can be executed to improve the performance of the data acquisition system, thereby improving the efficiency of data acquisition.
[0082] The process of determining whether the performance data is abnormal may include: performing weighted summation on CPU usage, memory usage, IO usage, and database connection number to determine an abnormal index. Then, when the abnormal index is greater than a preset threshold, the performance data is determined to be abnormal.
[0083] It is understandable that when the performance data is abnormal, it means that the data acquisition system may be stuck or have other phenomena that affect the data acquisition performance.
[0084] The process of executing the abnormality control program may be: suspending all active query operations at the time of abnormal performance data, or suspending the active query operation with the largest resource usage at the time of abnormal performance data. Afterwards, when the number of query operations executed simultaneously is less than a preset threshold, the suspended query operation is re-executed. Among them, the active query operation is the query operation being executed. This ensures the efficiency and reliability of data acquisition.
[0085] Alternatively, after executing the query operation corresponding to the data acquisition request and before executing the next query operation, if the performance data is abnormal, the query operation corresponding to the data acquisition request can be suspended. Then, the query operation corresponding to the data acquisition request is re-executed at a preset time interval. When the number of times the query operation is re-executed is greater than a preset threshold, it is determined that the data acquisition request has failed, and the query operation corresponding to the data acquisition request is stopped. This ensures the efficiency and reliability of data acquisition.
[0086] In this application, in response to receiving a data acquisition request, after determining the evaluation strategy and the target database based on the screening dimension in the data acquisition request, the data acquisition duration or data acquisition amount is determined based on the evaluation strategy, so as to determine the target time for executing the query operation corresponding to the data acquisition request based on the data acquisition duration or data acquisition amount, and then, while executing the query operation at the target time, the system is performance monitored to obtain performance data, and in the case of abnormal performance data, an abnormal control program is executed. Thus, during the query operation, the system is performance monitored to execute the abnormal control program in the case of abnormal performance data, thereby improving the performance of the data acquisition system. Thus, the efficiency and reliability of data acquisition are guaranteed.
[0087] Figure 4 A flowchart of a data acquisition method provided in an embodiment of the present application.
[0088] like Figure 4 As shown, the data acquisition method includes the following steps:
[0089] Step 401, in response to receiving a data acquisition request, determining an evaluation strategy and a target data table based on a screening dimension in the data acquisition request.
[0090] Step 402: Determine the data acquisition duration or data acquisition amount based on the evaluation strategy.
[0091] Step 403: Determine a target time for executing a query operation corresponding to the data acquisition request based on the data acquisition duration or the data acquisition amount.
[0092] Step 404, performing a query operation at a target time to obtain target data from a target data table.
[0093] In the present application, the specific implementation process of steps 401 to 404 can be found in the detailed description of any embodiment of the present application, and will not be repeated here.
[0094] Step 405, when the evaluation strategy or target data table fails to be determined based on the screening dimension in the data acquisition request, the query operation corresponding to the data acquisition request is executed, and the system performance is monitored to obtain performance data.
[0095] In this application, if the data table mapping information does not include the target data table corresponding to the screening dimension, the evaluation strategy is determined to fail. The evaluation strategy corresponding to the screening dimension is not set in the system, and the target data table is determined to fail.
[0096] In the event that the evaluation strategy or target data table fails to be determined based on the screening dimensions in the data acquisition request, the query operation corresponding to the data acquisition request is executed, and the system performance is monitored to obtain performance data.
[0097] Among them, the process of executing the query operation corresponding to the data acquisition request, and the process of monitoring the performance of the system and obtaining performance data can be referred to the detailed description of any embodiment of the present application, which will not be repeated here.
[0098] Step 406: When the performance data is abnormal, execute the abnormal control program.
[0099] In the present application, the specific implementation process of step 406 can be found in the detailed description of any embodiment of the present application, and will not be repeated here.
[0100] In this application, when the evaluation strategy or target data table fails to be determined based on the screening dimension in the data acquisition request, the query operation corresponding to the data acquisition request is executed, and the system is monitored for performance to obtain performance data. After that, when the performance data is abnormal, the abnormal control program is executed. In this way, the efficiency of data acquisition is guaranteed while the normal operation of data acquisition is guaranteed.
[0101] In order to implement the above embodiment, the present application also proposes a data acquisition device.
[0102] Figure 5 A schematic diagram of the structure of a data acquisition device provided in an embodiment of the present application.
[0103] like Figure 5 As shown, the data acquisition device includes a matching module 510 , a first determination module 520 , a second determination module 530 , and an execution module 540 .
[0104] A matching module 510 is used to determine an evaluation strategy and a target data table based on a screening dimension in the data acquisition request in response to receiving the data acquisition request;
[0105] A first determination module 520, configured to determine a data acquisition duration or a data acquisition amount based on an evaluation strategy;
[0106] A second determination module 530 is used to determine a target time for executing a query operation corresponding to a data acquisition request based on a data acquisition duration or a data acquisition amount;
[0107] The execution module 540 is used to execute a query operation at a target time to obtain target data from a target data table.
[0108] Furthermore, in a possible implementation of the embodiment of the present application, the matching module 510 is configured to:
[0109] When the current moment is within a preset time period, in response to receiving a data acquisition request, an evaluation strategy and a target data table are determined based on the screening dimensions in the data acquisition request, wherein the preset time period is a time period in which the total amount of historical data acquisition requests received within the historical time period is greater than a preset threshold.
[0110] Furthermore, in a possible implementation of the embodiment of the present application, the execution module 540 is configured to:
[0111] Perform system performance monitoring while executing query operations at the target time to obtain performance data, including CPU usage, memory usage, IO usage, and number of database connections.
[0112] In case of abnormal performance data, the abnormal control procedure is executed.
[0113] Furthermore, in a possible implementation of the embodiment of the present application, the execution module 540 is configured to:
[0114] Suspend active query operations at the time when performance data is abnormal, and re-execute active query operations when the number of query operations executed simultaneously is less than a preset threshold; or,
[0115] The query operation is paused, and the query operation is re-executed at a preset time interval. When the number of times the query operation is re-executed is greater than a threshold, it is determined that the data acquisition request has failed.
[0116] Furthermore, in a possible implementation of the embodiment of the present application, the execution module 540 is further configured to:
[0117] In the case where the evaluation strategy or target database fails to be determined based on the screening dimensions in the data acquisition request, the query operation corresponding to the data acquisition request is executed, and the system performance is monitored to obtain performance data;
[0118] In case of abnormal performance data, the abnormal control procedure is executed.
[0119] Furthermore, in a possible implementation of the embodiment of the present application, a feedback module is also included, which is used to:
[0120] If the target user's permission in the data acquisition request is approved, the target data is fed back to the target user;
[0121] If the target user's permission review fails, it is determined that the data acquisition request has failed.
[0122] Furthermore, in a possible implementation of the embodiment of the present application, a security module is also included, which is used to:
[0123] When the target data contains sensitive data, the sensitive data is masked.
[0124] It should be noted that the above explanations of the data acquisition method embodiment are also applicable to the data acquisition device of this embodiment, and will not be repeated here.
[0125] In this application, in response to receiving a data acquisition request, an evaluation strategy and a target data table are determined based on the screening dimension in the data acquisition request. After that, based on the evaluation strategy, the data acquisition duration or the data acquisition amount is determined, and based on the data acquisition duration or the data acquisition amount, the target time for executing the query operation corresponding to the data acquisition request is determined, and then the query operation is executed at the target time to obtain the target data from the target data table. Thus, by automatically determining and executing the query operation corresponding to the data acquisition request, the process of manually writing the data acquisition script is omitted. At the same time, according to the data acquisition duration or the data acquisition amount, the timing (target time) of executing the query operation is determined to effectively avoid resource competition. Thereby improving the efficiency of data acquisition.
[0126] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.
[0127] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.
[0128] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.
[0129] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.
[0130] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.
[0131] This application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, this application is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, risks can be minimized by limiting data collection and deleting data. In addition, when applicable, personal identifiers are removed from such personal information to protect the privacy of users.
[0132] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0133] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0134] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0135] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.
[0136] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0137] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0138] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0139] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A data acquisition method, It is characterized in that The method comprises: In response to receiving a data acquisition request, determining an evaluation strategy and a target data table based on a screening dimension in the data acquisition request; Based on the evaluation strategy, determine the data acquisition time or data acquisition amount; Based on the data acquisition duration or the data acquisition amount, determining a target time for executing a query operation corresponding to the data acquisition request; The query operation is performed at the target time to obtain target data from the target data table.
2. The method according to claim 1, It is characterized in that In response to receiving the data acquisition request, the evaluation strategy is matched based on the screening dimension in the data acquisition request, including: When the current moment is within a preset time period, in response to receiving a data acquisition request, an evaluation strategy and a target data table are determined based on the screening dimensions in the data acquisition request, wherein the preset time period is a time period in which the total amount of historical data acquisition requests received within the historical time period is greater than a preset threshold.
3. The method according to claim 1, It is characterized in that The performing the query operation at the target time to obtain the target data includes: Performing performance monitoring on the system while executing the query operation at the target time to obtain performance data, wherein the performance data includes CPU usage, memory usage, IO usage, and the number of database connections; In the case where the performance data is abnormal, an abnormality control program is executed.
4. The method according to claim 3, It is characterized in that The abnormal control procedure includes: Suspending the active query operation at the time when the performance data is abnormal, and re-executing the active query operation when the number of query operations executed simultaneously is less than a preset threshold; or, The query operation is paused, and the query operation is re-executed at a preset time interval, and when the number of times the query operation is re-executed is greater than a threshold, it is determined that the data acquisition request has failed.
5. The method according to claim 1, It is characterized in that Also includes: In the case where the evaluation strategy or the target data table fails to be determined based on the screening dimension in the data acquisition request, executing the query operation corresponding to the data acquisition request, and performing performance monitoring on the system to obtain performance data; In the case where the performance data is abnormal, an abnormality control program is executed.
6. The method according to claim 1, It is characterized in that Also includes: If the target user's authority in the data acquisition request is approved, the target data is fed back to the target user; When the authority review of the target user fails, it is determined that the data acquisition request has failed.
7. The method according to claim 1, It is characterized in that Also includes: In the case where the target data contains sensitive data, mask processing is performed on the sensitive data.
8. A data acquisition device, It is characterized in that The device comprises: A matching module, configured to, in response to receiving a data acquisition request, determine an evaluation strategy and a target data table based on a screening dimension in the data acquisition request; A first determination module, used to determine the data acquisition duration or data acquisition amount based on the evaluation strategy; A second determination module, configured to determine a target time for executing a query operation corresponding to the data acquisition request based on the data acquisition duration or the data acquisition amount; An execution module is used to execute the query operation at the target time to obtain target data from the target data table.
9. The device as claimed in claim 8, It is characterized in that The matching module is used for: When the current moment is within a preset time period, in response to receiving a data acquisition request, an evaluation strategy and a target data table are determined based on the screening dimensions in the data acquisition request, wherein the preset time period is a time period in which the total amount of historical data acquisition requests received within the historical time period is greater than a preset threshold.
10. The device according to claim 8, It is characterized in that The execution module is used to: Performing performance monitoring on the system while executing the query operation at the target time to obtain performance data, wherein the performance data includes CPU usage, memory usage, IO usage, and the number of database connections; In the case where the performance data is abnormal, an abnormality control program is executed.
11. The device according to claim 10, It is characterized in that The execution module is used to: Suspending the active query operation at the time when the performance data is abnormal, and re-executing the active query operation when the number of query operations executed simultaneously is less than a preset threshold; or, The query operation is paused, and the query operation is re-executed at a preset time interval, and when the number of times the query operation is re-executed is greater than a threshold, it is determined that the data acquisition request has failed.
12. The device according to claim 8, It is characterized in that The execution module is also used to: In the case where the evaluation strategy or the target data table fails to be determined based on the screening dimension in the data acquisition request, executing the query operation corresponding to the data acquisition request, and performing performance monitoring on the system to obtain performance data; In the case where the performance data is abnormal, an abnormality control program is executed.
13. The device according to claim 8, It is characterized in that Also included are feedback modules for: If the target user's authority in the data acquisition request is approved, the target data is fed back to the target user; When the authority review of the target user fails, it is determined that the data acquisition request has failed.
14. The device according to claim 8, It is characterized in that Also included are security modules for: In the case where the target data contains sensitive data, mask processing is performed on the sensitive data.
15. An electronic device, It is characterized in that include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.
16. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.
17. A computer program product, It is characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when being executed by a processor.
Citation Information
Patent Citations
Data inquiring method and data inquiring server
CN102081626A
Data query method and device
CN110334118A
Data access proxy method and device, computer equipment and storage medium
CN118194258A
Data query system load balancing
US20050010558A1
Cited By
Request processing method and device, electronic equipment, storage medium and program product
CN119759970A
