Data acquisition method, electronic device, storage medium, and program product

Abstract

The present disclosure relates to a data acquisition method, an electronic device, a storage medium and a program product. The method comprises: acquiring a target data acquisition request, the target data acquisition request being used to indicate a target data acquisition index and an initial parameter corresponding to the target data acquisition index; acquiring a target arrangement configuration corresponding to the target data acquisition index from an index arrangement configuration library, the index arrangement configuration library comprising a plurality of data acquisition indexes and an arrangement configuration corresponding to each data acquisition index; the target arrangement configuration being used to indicate a logical execution relationship of at least one data acquisition node corresponding to the target data acquisition index, the at least one data acquisition node respectively corresponding to a plurality of data sources; and based on the initial parameter, executing the at least one data acquisition node according to the target arrangement configuration to acquire a target data result corresponding to the target data acquisition request.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a data acquisition method, electronic device, storage medium, and program product. Background Technology

[0002] The scenario of dynamically acquiring data from multiple data sources is widely found in fields such as enterprise digital operations, public services, and scientific research analysis. The core is to integrate scattered and heterogeneous data sources (such as databases, API interfaces, file systems, IoT devices, etc.) to acquire information in real time or near real time to support decision-making, business processing, or service delivery.

[0003] For example, some residential service platforms require dynamic data acquisition from multiple data sources for scenarios such as dynamic verification and updating of housing information, matching user needs and optimizing services, transaction processes and performance guarantees, and industry data and market analysis. Traditional methods of dynamically acquiring data from multiple data sources typically rely on hard coding, which not only limits system flexibility but also increases maintenance costs. With rapidly changing business requirements, developers need to frequently modify code to adapt to new data acquisition logic, leading to decreased development efficiency and increased errors. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this disclosure provides a data acquisition method, apparatus, electronic device, storage medium and program product.

[0005] A first aspect of this disclosure provides a data acquisition method, the method comprising: acquiring a target data acquisition request, the target data acquisition request indicating a target data acquisition indicator and initial parameters corresponding to the target data acquisition indicator; acquiring a target orchestration configuration corresponding to the target data acquisition indicator from an indicator orchestration configuration library; the indicator orchestration configuration library including multiple data acquisition indicators and orchestration configurations corresponding to each data acquisition indicator; the target orchestration configuration indicating the logical execution relationship of at least one data acquisition node corresponding to the target data acquisition indicator, the at least one data acquisition node corresponding to multiple data sources respectively; and executing the at least one data acquisition node according to the target orchestration configuration based on the initial parameters to acquire a target data result corresponding to the target data acquisition request.

[0006] In some embodiments of this disclosure, the execution of at least one data acquisition node based on the initial parameters and according to the target orchestration configuration to acquire the target data result corresponding to the target data acquisition request includes:

[0007] A local parameter pool is constructed, which includes the initial parameter; the following steps S1 are executed for each of the at least one data acquisition node until the at least one data acquisition node completes execution, and the target data result is obtained from the local parameter pool; wherein, step S1 includes: obtaining the target input parameter corresponding to the target data acquisition node from the local parameter pool, the target input parameter being determined based on at least one of the following: the initial parameter, the output parameter corresponding to the output parameter obtained by the data acquisition node executed before the target data acquisition node; obtaining the target output parameter from the data source corresponding to the target data acquisition node based on the target input parameter; and saving the target output parameter to the local parameter pool.

[0008] In some embodiments of this disclosure, obtaining target output parameters from the data source corresponding to the target data acquisition node based on the target input parameters includes: obtaining target response data from the data source corresponding to the target data acquisition node based on the target input parameters; and parsing the target output parameters from the target response data based on the output data model corresponding to the target data acquisition node.

[0009] In some embodiments of this disclosure, when the data volume of the target input parameter is greater than or equal to a data volume threshold, the target input parameter is divided into multiple sub-input parameters, and the data volume of each sub-input parameter is less than the data volume threshold; based on the multiple sub-input parameters, the corresponding sub-output parameters are sequentially obtained from the data source corresponding to the target data acquisition node to obtain the target output parameter.

[0010] In some embodiments of this disclosure, the method of parsing the target output parameter from the target response data based on the output data model corresponding to the target data acquisition node includes: parsing a first output parameter that matches the data structure indicated by the output data model from the target response data based on the output data model corresponding to the target data acquisition node; selecting a second output parameter that satisfies the filtering conditions indicated by the output filtering expression from the first output parameter based on the output filtering expression corresponding to the target data acquisition node; and performing type conversion on the second output parameter to obtain the target output parameter.

[0011] In some embodiments of this disclosure, obtaining the target input parameter corresponding to the target data acquisition node from the local parameter pool includes: converting the first input parameter obtained from the local parameter pool into a second input parameter that matches the data structure indicated by the parameter pool value model based on the parameter pool value model corresponding to the target data acquisition node; and filtering out the target input parameter that satisfies the filtering conditions indicated by the input filtering expression from the second input parameter based on the input filtering expression corresponding to the target data acquisition node.

[0012] In some embodiments of this disclosure, the method further includes: obtaining predefined fixed value parameters based on the transparent fixed value model corresponding to the target data acquisition node; obtaining target output parameters from the data source corresponding to the target data acquisition node based on the target input parameters includes: obtaining the target output parameters from the data source corresponding to the target data acquisition node based on the target input parameters and the fixed value parameters.

[0013] In some embodiments of this disclosure, obtaining the target data result from the local parameter pool includes: obtaining all parameters from the local parameter pool; and performing result arrangement processing on all parameters to obtain the target data result.

[0014] In some embodiments of this disclosure, the method further includes: when the execution of the first data acquisition node in the at least one data acquisition node is abnormally interrupted, outputting abnormal interruption information corresponding to the first data acquisition node; the abnormal interruption information includes at least one of the following: abnormal interruption status, abnormal interruption reason, and local result data corresponding to the data acquisition node that was executed before the abnormal interruption.

[0015] A second aspect of this disclosure provides a data acquisition apparatus, comprising: an acquisition module configured to acquire a target data acquisition request, the target data acquisition request indicating a target data acquisition indicator and initial parameters corresponding to the target data acquisition indicator; acquire a target orchestration configuration corresponding to the target data acquisition indicator from an indicator orchestration configuration library; the indicator orchestration configuration library includes multiple data acquisition indicators and orchestration configurations corresponding to each data acquisition indicator; the target orchestration configuration indicates the logical execution relationship of at least one data acquisition node corresponding to the target data acquisition indicator, the at least one data acquisition node corresponding to multiple data sources; and execute the at least one data acquisition node according to the target orchestration configuration based on the initial parameters to acquire a target data result corresponding to the target data acquisition request.

[0016] A third aspect of this disclosure provides an electronic device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the data acquisition method as described in the first aspect.

[0017] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data acquisition method as described in the first aspect.

[0018] A fifth aspect of this disclosure provides a computer program product, wherein the computer program product includes a computer program that, when the computer program product is run on a processor, causes the processor to execute the computer program to implement the data acquisition method as described in the first aspect.

[0019] A sixth aspect of this disclosure provides a chip including a processor and a communication interface coupled to the processor, the processor being used to execute program instructions to implement the data acquisition method as described in the first aspect.

[0020] The technical solution provided in this disclosure has the following advantages compared with the prior art: A target data acquisition request is obtained, which indicates a target data acquisition indicator and the initial parameters corresponding to the target data acquisition indicator; a target orchestration configuration corresponding to the target data acquisition indicator is obtained from an indicator orchestration configuration library; the indicator orchestration configuration library includes multiple data acquisition indicators and orchestration configurations corresponding to each data acquisition indicator; the target orchestration configuration indicates the logical execution relationship of at least one data acquisition node corresponding to the target data acquisition indicator, and the at least one data acquisition node corresponds to multiple data sources; based on the initial parameters, the at least one data acquisition node is executed according to the target orchestration configuration to obtain the target data result corresponding to the target data acquisition request. In this way, by configuring the metric orchestration configuration library, the data acquisition metrics corresponding to the data acquisition request are associated and mapped with the orchestration configuration corresponding to the data acquisition request. This decouples hard coding from the data acquisition logic. By associating the data acquisition logic with a unique data acquisition metric, the dynamic definition and flexible expansion of the data acquisition logic are realized, avoiding hard coding. Business personnel can quickly adjust the association between the data acquisition metrics and the orchestration configuration of the data acquisition logic by modifying the metric orchestration configuration library, without the need for developers to frequently modify the code based on changes in the data acquisition logic. The data acquisition logic can be flexibly adjusted without modifying the code, improving the maintainability and scalability of the system, reducing the system maintenance cost, supporting rapid business iteration, improving development efficiency, and reducing the error rate of the code. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0022] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is one of the flowcharts illustrating a data acquisition method provided in an embodiment of this disclosure;

[0024] Figure 2 A second schematic flowchart illustrating the data acquisition method provided in this embodiment of the disclosure;

[0025] Figure 3 The third schematic flowchart of the data acquisition method provided in this embodiment of the disclosure;

[0026] Figure 4 A structural block diagram of a data acquisition device provided in an embodiment of this disclosure;

[0027] Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0028] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0029] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0030] The terms "first," "second," etc., used in this disclosure and in the claims are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this disclosure can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0031] The electronic devices in this disclosure can be mobile electronic devices or non-mobile electronic devices. Mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc.; non-mobile electronic devices can be personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc.; this disclosure does not impose specific limitations.

[0032] The execution subject of the data acquisition method provided in this disclosure can be the aforementioned electronic device (including mobile electronic devices and non-mobile electronic devices), or it can be a functional module and / or functional entity in the electronic device that can implement the data acquisition method. The specific implementation subject can be determined according to actual usage requirements, and this disclosure does not limit it.

[0033] The data acquisition method provided in this disclosure will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0034] like Figure 1 As shown in the embodiments of this disclosure, a data acquisition method is provided, which may include the following steps 101 to 103.

[0035] 101. Obtain the target data acquisition request. This target data acquisition request is used to indicate the target data acquisition indicator and the initial parameters corresponding to the target data acquisition indicator.

[0036] Among them, data acquisition metrics usually refer to the metric code or metric identifier for data acquisition requests. It is a unique and standardized string identifier corresponding to the data acquisition request, a technical abbreviation or code of the business logic corresponding to the data acquisition request, and a core bridge connecting business metrics and technical implementation.

[0037] In some embodiments of this disclosure, the target data acquisition request may directly carry the target data acquisition indicators and initial parameters, or the target data acquisition indicators and initial parameters indicated by the target data acquisition request may be determined by parsing the target data acquisition request. The specific method can be determined according to the actual situation and is not limited here.

[0038] 102. Retrieve the target orchestration configuration corresponding to the target data acquisition indicator from the indicator orchestration configuration library.

[0039] The metric orchestration configuration library includes multiple data acquisition metrics and orchestration configurations corresponding to each data acquisition metric. The target orchestration configuration is used to indicate the logical execution relationship of at least one data acquisition node corresponding to the target data acquisition metric, and the at least one data acquisition node corresponds to multiple data sources.

[0040] The orchestration configuration may include at least one data acquisition node corresponding to the data acquisition metric, the execution order of at least one data acquisition node, the sequential dependencies, the execution logic of each data acquisition node (such as data acquisition rules, data filtering conditions, and data association methods), parameter sources, parameter input and output formats, and may also include other information, which is not limited here.

[0041] In the indicator orchestration configuration library, there is a one-to-one correspondence between multiple data acquisition indicators and multiple orchestration configurations, with each data acquisition indicator uniquely corresponding to one orchestration configuration.

[0042] The logical execution relationship can include parallel execution relationship, serial execution relationship, or a combination of both; no limitation is made here.

[0043] Different data acquisition nodes can correspond to the same data source or different data sources. That is, at least one data acquisition node can have a one-to-one correspondence with multiple data sources or a many-to-one correspondence. The specific correspondence can be determined according to the actual situation and is not limited here.

[0044] The data acquisition node includes the request protocol (method), request path (path), request input parameters, request output parameters, IAM verification information, etc., which can be determined according to the actual situation and are not limited here. The input parameter information can include one or more input parameters, specifically including the name of each input parameter, the extractor for the input parameter, the data model used by the extractor, and the data source (datas), etc., which are not limited here. The output parameter information can include one or more output parameters, specifically including the name of each output parameter, the extractor for the output parameter, the data model used by the extractor, and the data storage format, etc., which are not limited here.

[0045] In some embodiments of this disclosure, an indicator orchestration configuration library is accessed through an SPI plugin extension mechanism and / or a default configuration file mode. This facilitates the retrieval of the target orchestration configuration corresponding to the target data acquisition indicator from the indicator orchestration configuration library. Thus, access to multiple data sources is supported. Through SPI plugin extension, the IDataReadingStrategy interface supports non-HTTP protocols (such as database queries and RPC calls). The SPI mechanism flexibly extends protocol support to adapt to data sources with different technology stacks. Loading through the default configuration file mode means that modifications to resource / url_config.properties do not require a service restart and take effect dynamically.

[0046] 103. Based on the initial parameters, execute at least one data acquisition node according to the target orchestration configuration to obtain the target data result corresponding to the target data acquisition request.

[0047] The target data result is the final output result corresponding to the target data acquisition request. It can be the data output result of the last data acquisition node executed in at least one data acquisition node, or it can be the output result obtained by filtering, combining and other processing of the data output results generated during the execution of at least one data acquisition node. There is no limitation here.

[0048] In this embodiment, by configuring an indicator orchestration configuration library, the data acquisition indicator corresponding to the data acquisition request is associated and mapped with the orchestration configuration corresponding to the data acquisition request. This decouples hard coding from the data acquisition logic. By associating the data acquisition logic with a unique data acquisition indicator, the dynamic definition and flexible expansion of the data acquisition logic are realized, avoiding hard coding. Business personnel can quickly adjust the association between the data acquisition indicator and the orchestration configuration of the data acquisition logic by modifying the indicator orchestration configuration library. There is no need for development personnel to frequently modify the code based on changes in the data acquisition logic. The data acquisition logic can be flexibly adjusted without modifying the code, which improves the maintainability and scalability of the system, reduces the system maintenance cost, supports rapid business iteration, improves development efficiency, and reduces the error rate of the code.

[0049] In some embodiments of this disclosure, the orchestration configuration may include data flow information between data acquisition nodes that have sequential dependencies, such as which data acquisition nodes' outputs are included in the input of a data acquisition node, and which data acquisition nodes' inputs are the output of that data acquisition node.

[0050] In some embodiments of this disclosure, combined with Figure 1 ,like Figure 2 As shown, step 103 can be implemented through steps 103a to 103b.

[0051] 103a. Construct a local parameter pool, which includes the initial parameter.

[0052] 103b. Perform the following steps S1 for each of the at least one data acquisition node until the at least one data acquisition node completes its execution and obtains the target data result from the local parameter pool.

[0053] Among them, such as Figure 3 As shown, step S1 includes the following steps S11 to S13.

[0054] S11. Obtain the target input parameters corresponding to the target data acquisition node from the local parameter pool.

[0055] The target input parameter is determined based on at least one of the following: the initial parameter, the output parameter corresponding to the output parameter obtained by the data acquisition node before the target data acquisition node;

[0056] S12. Based on the target input parameters, obtain the target output parameters from the data source corresponding to the target data acquisition node.

[0057] S13. Save the target output parameters to the local parameter pool.

[0058] It is understood that step S1 is executed for each of the at least one data acquisition nodes. That is, for each data acquisition node, input parameters are obtained from the local parameter pool. Based on the input parameters, the corresponding output parameters are obtained from the data source corresponding to the target data acquisition node. The output parameters are stored in the local parameter pool as input parameters for subsequent data acquisition nodes or for determining the target data result. At least one data acquisition node is executed serially and / or in parallel according to the orchestration configuration until all data acquisition nodes have been executed and the target data result is obtained from the local parameter pool.

[0059] In this embodiment of the disclosure, data sharing and flow between data acquisition nodes are realized through a parameter pool, which simplifies the data acquisition process. Among data acquisition nodes with sequential dependencies, the output of the previous data acquisition node is automatically used as the input of the subsequent data acquisition node, which simplifies the data acquisition process and supports multi-node chain calls (such as interface A result → processing → interface B input), meeting the needs of multi-level data processing.

[0060] In some embodiments of this disclosure, step S12 can be specifically implemented by the following steps S12a and S12b.

[0061] S12a. Based on the target input parameters, obtain the target response data from the data source corresponding to the target data acquisition node.

[0062] S12b: Based on the output data model corresponding to the target data acquisition node, the target output parameters are parsed from the target response data.

[0063] The output data model can include any of the following: JSONPATH_MODE (JSONPath parsing), JSONPATH_MAP_LIST_MODE (key-value pair list modeling), and other data models, which are not limited here.

[0064] Among them, the output data model is a standardized template that defines the structure, format, and field rules of the final output data. Its core function is to ensure that the output data structure is consistent, the meaning is clear, and it can be correctly parsed and used by downstream systems, thus solving the problems of chaotic data output and difficulties in upstream and downstream integration.

[0065] In this embodiment of the disclosure, the target response data is parsed according to the rules corresponding to the output data model to obtain target output parameters, which are then stored in a local parameter pool so as to serve as input parameters for subsequent data acquisition nodes or to determine the target data acquisition result.

[0066] In some embodiments of this disclosure, step S12b can be specifically implemented by the following steps S12b1 to S12b3.

[0067] S12b1. Based on the output data model corresponding to the target data acquisition node, the first output parameter matching the data structure indicated by the output data model is parsed from the target response data.

[0068] S12b2. Based on the output filtering expression corresponding to the target data acquisition node, select the second output parameter from the first output parameter that meets the filtering conditions indicated by the output filtering expression.

[0069] The output filtering expression can be determined based on the actual situation and is not limited here. For example, business logic filtering can be implemented using JSONPath syntax without hard-coding conditional judgments.

[0070] S12b3. Convert the type of the second output parameter to obtain the target output parameter.

[0071] Type conversion can include TO_LIST (to list), TO_STRING (to string, with delimiter support), etc., and is not limited here.

[0072] Type conversion is a key step in the "standardized data flow" process of data processing. Its core function is to solve the problem of "data type mismatch", ensure that the data stored in the parameter pool can be correctly reused in subsequent stages, and improve the universality and security of the data.

[0073] In this embodiment of the disclosure, appropriate output parameters can be obtained by filtering expressions and type conversion, and then stored in a local parameter pool for easy subsequent use.

[0074] In some embodiments of this disclosure, step S12 can be specifically implemented by the following steps S12c and S12d.

[0075] S12c: If the data volume of the target input parameter is greater than or equal to the data volume threshold, the target input parameter is divided into multiple sub-input parameters, and the data volume of each sub-input parameter is less than the data volume threshold.

[0076] The data volume threshold can be determined based on the actual situation and is not limited here.

[0077] S12d: Based on the multiple sub-input parameters, obtain the corresponding sub-output parameters from the data source corresponding to the target data acquisition node in sequence to obtain the target output parameters.

[0078] After obtaining the response corresponding to each sub-input parameter, the corresponding sub-output parameters can be obtained by parsing the output data model. Then, the multiple sub-output parameters corresponding to the multiple sub-input parameters are merged to obtain the target output parameter.

[0079] In this embodiment of the disclosure, by setting a data volume threshold, the amount of data requested in a single request is controlled. When the data volume of the target input parameter is greater than or equal to the data volume threshold, the target input parameter obtained from the parameter pool is automatically split into multiple batches of input parameters (multiple sub-input parameters) according to the data volume threshold. This enables the sequential sending of POST requests or HTTP requests and the batching of data acquisition (sequentially obtaining the corresponding sub-output parameters from the data source corresponding to the target data acquisition node to obtain the target output parameter). This can avoid interface timeouts and improve system stability.

[0080] In some embodiments of this disclosure, step S11 can be specifically implemented by the following steps S11a and S11b.

[0081] S11a. Based on the parameter pool value model corresponding to the target data acquisition node, the first input parameter obtained from the local parameter pool is transformed into a second input parameter that matches the data structure indicated by the parameter pool value model.

[0082] The parameter pool value model can be SIMPLE_MODE (parameter pool value), or other parameter pool value models, which are not limited here.

[0083] S11b: Based on the input filtering expression corresponding to the target data acquisition node, select the target input parameters from the second input parameters that meet the filtering conditions indicated by the input filtering expression.

[0084] The input filtering expression can be determined based on the actual situation and is not limited here. For example, business logic filtering can be implemented using JSONPath syntax without hard-coding conditional judgments.

[0085] In this embodiment of the disclosure, input parameters are obtained from the local parameter pool according to the rules corresponding to the parameter pool value model. Suitable target input parameters can be obtained by filtering through filtering expressions, which are then used by the data acquisition node to make data requests.

[0086] In some embodiments of this disclosure, step S1 may further include step S14, and step S12 may be implemented by step S12e.

[0087] S14. Based on the target data acquisition node, obtain the transparent fixed value model and obtain the predefined fixed value parameters.

[0088] S12e, Based on the target input parameters, the target output parameters are obtained from the data source corresponding to the target data acquisition node, including:

[0089] Based on the target input parameters and the fixed value parameters, the target output parameters are obtained from the data source corresponding to the target data acquisition node.

[0090] In this embodiment of the disclosure, the input data model includes a parameter pool value model and a pass-through fixed value model. The input data model is a standardized template that defines the "structure, format, and verification rules of upstream incoming data". Its core function is to ensure that the input data is "legal, complete, and meets the downstream processing requirements" from the source, and to avoid downstream process abnormalities due to data problems. It is the "first line of defense" for data flow.

[0091] In this embodiment of the disclosure, by using the parameter pool value model and the transparent fixed value model, the input parameters of the data acquisition node can be acquired more flexibly and diversely, thereby improving the efficiency of the input parameters.

[0092] In some embodiments of this disclosure, the above steps of obtaining the target data result from the local parameter pool can be achieved by obtaining all parameters from the local parameter pool and performing result arrangement processing on all parameters to obtain the target data result.

[0093] Among these, result arrangement processing is a crucial step in restructuring, logically associating, formatting, and optimizing the parameter extraction results from the local result pool. It transforms scattered and isolated data into usable and standardized data that meets business needs, serving as a core bridge connecting data extraction and business applications.

[0094] The result arrangement and processing can include data association (establishing logical relationships), format conversion (unifying data standards), content cleaning (improving data quality), and structure reorganization (adapting to downstream needs).

[0095] This includes data association and the aggregation of data from multiple sources through primary keys (such as ucId) to form business relationships.

[0096] The format conversion includes type conversion (e.g., converting the string "1001" to the numeric type 1001 to ensure that downstream calculations are correct), format adjustment (e.g., converting the timestamp 1716201600 to "2024-05-2100:00:00" for easier front-end display), and unit unification (e.g., converting the amount from "yuan" 99.9 to "fen" 9990 to adapt to the payment system requirements).

[0097] Content cleaning includes filtering (such as removing records with empty ucId or invalid orgCode), deduplication (such as merging duplicate userName information), and completion (such as filling empty userName with the default value "unknown" or supplementing the user's mobile phone number by calling the interface through ucId).

[0098] The structural reorganization includes converting lists to key-value pairs (e.g., converting [{"id":1,"name":"a"},{"id":2,"name":"b"}] to {1:"a",2:"b"} for faster querying), flattening nesting (e.g., converting {"user":{"info":{"name":"Zhang San"}}} to {"user_name":"Zhang San"} for simplified downstream calls), and splitting / merging fields (e.g., splitting "Zhang San-1001" into name:"Zhang San" and ucId:1001).

[0099] In this embodiment of the disclosure, result arrangement processing can adapt the target data result to the target data acquisition request, thereby improving data acquisition efficiency.

[0100] In some embodiments of this disclosure, the data acquisition method provided in the embodiments of this disclosure may further include the following step 104.

[0101] 104. If the execution of the first data acquisition node in at least one data acquisition node is abnormally interrupted, output the abnormal interruption information corresponding to the first data acquisition node.

[0102] The abnormal interruption information includes at least one of the following: abnormal interruption status, abnormal interruption reason, and local result data corresponding to the data acquisition node that was executed before the abnormal interruption.

[0103] The abnormal interruption status can include which data acquisition node the interruption occurred at, and the abnormal interruption cause can include the reason that caused the abnormal interruption.

[0104] In this embodiment of the disclosure, during the execution of the at least one data acquisition node, abnormal interruption information can be output to prompt the user that an interruption has occurred and related interruption information, which facilitates the user to retry or debug the interruption.

[0105] Figure 4 This is a structural block diagram of a data acquisition device shown in an embodiment of the present disclosure, such as... Figure 4 As shown, it includes: an acquisition module 401, used to acquire a target data acquisition request, the target data acquisition request indicating a target data acquisition indicator and initial parameters corresponding to the target data acquisition indicator; acquire a target orchestration configuration corresponding to the target data acquisition indicator from an indicator orchestration configuration library; the indicator orchestration configuration library includes multiple data acquisition indicators and orchestration configurations corresponding to each data acquisition indicator; the target orchestration configuration indicates the logical execution relationship of at least one data acquisition node corresponding to the target data acquisition indicator, the at least one data acquisition node corresponding to multiple data sources respectively; based on the initial parameters, execute the at least one data acquisition node according to the target orchestration configuration to acquire the target data result corresponding to the target data acquisition request.

[0106] In some embodiments of this disclosure, the acquisition module 401 is specifically used to construct a local parameter pool, which includes the initial parameter; and to execute the following step S1 for each of the at least one data acquisition node until the at least one data acquisition node has completed execution, and to acquire the target data result from the local parameter pool; wherein, step S1 includes: acquiring the target input parameter corresponding to the target data acquisition node from the local parameter pool, the target input parameter being determined based on at least one of the following: the initial parameter, the output parameter corresponding to the output parameter obtained by the data acquisition node executed before the target data acquisition node; acquiring the target output parameter from the data source corresponding to the target data acquisition node based on the target input parameter; and saving the target output parameter to the local parameter pool.

[0107] In some embodiments of this disclosure, the acquisition module 401 is specifically used to acquire target response data from the data source corresponding to the target data acquisition node based on the target input parameters; and to parse the target output parameters from the target response data based on the output data model corresponding to the target data acquisition node.

[0108] In some embodiments of this disclosure, the acquisition module 401 is specifically used to divide the target input parameter into multiple sub-input parameters when the data volume of the target input parameter is greater than or equal to a data volume threshold, wherein the data volume of each sub-input parameter is less than the data volume threshold; and based on the multiple sub-input parameters, sequentially acquire the corresponding sub-output parameters from the data source corresponding to the target data acquisition node to obtain the target output parameter.

[0109] In some embodiments of this disclosure, the acquisition module 401 is specifically used to: parse the target response data based on the output data model corresponding to the target data acquisition node to obtain a first output parameter that matches the data structure indicated by the output data model; filter out a second output parameter that satisfies the filtering conditions indicated by the output filtering expression based on the output filtering expression corresponding to the target data acquisition node; and perform type conversion on the second output parameter to obtain the target output parameter.

[0110] In some embodiments of this disclosure, the acquisition module 401 is specifically used to convert a first input parameter obtained from the local parameter pool into a second input parameter that matches the data structure indicated by the parameter pool value model based on the parameter pool value model corresponding to the target data acquisition node; and to filter out the target input parameter that satisfies the filtering conditions indicated by the input filtering expression from the second input parameter based on the input filtering expression corresponding to the target data acquisition node.

[0111] In some embodiments of this disclosure, the acquisition module 401 is further configured to acquire predefined fixed value parameters based on the transparent fixed value model corresponding to the target data acquisition node; specifically, the acquisition module 401 is configured to acquire the target output parameters from the data source corresponding to the target data acquisition node based on the target input parameters and the fixed value parameters.

[0112] In some embodiments of this disclosure, the acquisition module 401 is specifically used to acquire all parameters from the local parameter pool; and to perform result arrangement processing on all parameters to obtain the target data result.

[0113] In some embodiments of this disclosure, the device further includes: an output module 402, configured to output abnormal interruption information corresponding to the first data acquisition node when the execution of the first data acquisition node in the at least one data acquisition node is abnormally interrupted; the abnormal interruption information includes at least one of the following: abnormal interruption status, abnormal interruption reason, and local result data corresponding to the data acquisition node that was executed before the abnormal interruption.

[0114] In this embodiment, each module can implement the data acquisition method provided in the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0115] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. It is used to illustrate the electronic device that implements any data acquisition method in the embodiments of the present disclosure and should not be construed as a specific limitation on the embodiments of the present disclosure.

[0116] like Figure 5 As shown, the electronic device 500 may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processor 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0117] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although an electronic device 500 with various devices is shown, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0118] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processor 501, it can perform the functions defined in any data acquisition method provided in embodiments of this disclosure.

[0119] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0120] In some implementations, the client and server can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0121] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0122] The aforementioned computer-readable medium carries one or more programs. When the electronic device executes the aforementioned one or more programs, the electronic device causes the following to occur: An acquisition module is configured to acquire a target data acquisition request, the target data acquisition request indicating a target data acquisition indicator and initial parameters corresponding to the target data acquisition indicator; acquire a target orchestration configuration corresponding to the target data acquisition indicator from an indicator orchestration configuration library; the indicator orchestration configuration library includes multiple data acquisition indicators and orchestration configurations corresponding to each data acquisition indicator; the target orchestration configuration indicates the logical execution relationship of at least one data acquisition node corresponding to the target data acquisition indicator, the at least one data acquisition node corresponding to multiple data sources; and, based on the initial parameters, execute the at least one data acquisition node according to the target orchestration configuration to acquire the target data result corresponding to the target data acquisition request.

[0123] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on a computer, partially on a computer, as a standalone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0125] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0126] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0127] In the context of this disclosure, a computer-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0128] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0129] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0130] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A data acquisition method, characterized in that, The method includes: Obtain a target data acquisition request, wherein the target data acquisition request is used to indicate the target data acquisition indicator and the initial parameters corresponding to the target data acquisition indicator; From the indicator orchestration configuration library, obtain the target orchestration configuration corresponding to the target data acquisition indicator; the indicator orchestration configuration library includes multiple data acquisition indicators and orchestration configurations corresponding to each data acquisition indicator; the target orchestration configuration is used to indicate the logical execution relationship of at least one data acquisition node corresponding to the target data acquisition indicator, and the at least one data acquisition node corresponds to multiple data sources respectively; Construct a local parameter pool, which includes the initial parameters; For each of the at least one data acquisition node, the following step S1 is executed until the at least one data acquisition node completes its execution, and the target data result is obtained from the local parameter pool. Step S1 includes: The target input parameters corresponding to the target data acquisition node are obtained from the local parameter pool. The target input parameters are determined based on at least one of the following: the initial parameters, and the output parameters corresponding to the output parameters obtained by executing the data acquisition node before the target data acquisition node. Based on the target input parameters, obtain the target output parameters from the data source corresponding to the target data acquisition node; The target output parameters are saved to the local parameter pool.

2. The method according to claim 1, characterized in that, The step of obtaining target output parameters from the data source corresponding to the target data acquisition node based on the target input parameters includes: Based on the target input parameters, target response data is obtained from the data source corresponding to the target data acquisition node; Based on the output data model corresponding to the target data acquisition node, the target output parameters are parsed from the target response data; And / or, If the data volume of the target input parameter is greater than or equal to the data volume threshold, the target input parameter is divided into multiple sub-input parameters, and the data volume of each sub-input parameter is less than the data volume threshold. Based on the multiple sub-input parameters, the corresponding sub-output parameters are sequentially obtained from the data source corresponding to the target data acquisition node to obtain the target output parameters.

3. The method according to claim 2, characterized in that, The method of parsing the target output parameters from the target response data based on the output data model corresponding to the target data acquisition node includes: Based on the output data model corresponding to the target data acquisition node, a first output parameter matching the data structure indicated by the output data model is parsed from the target response data; Based on the output filtering expression corresponding to the target data acquisition node, a second output parameter that meets the filtering conditions indicated by the output filtering expression is selected from the first output parameter; The second output parameter is converted to a different type to obtain the target output parameter.

4. The method according to claim 1, characterized in that, The step of obtaining the target input parameters corresponding to the target data acquisition node from the local parameter pool includes: Based on the parameter pool value model corresponding to the target data acquisition node, the first input parameter obtained from the local parameter pool is transformed into a second input parameter that matches the data structure indicated by the parameter pool value model. Based on the input filtering expression corresponding to the target data acquisition node, the target input parameters that satisfy the filtering conditions indicated by the input filtering expression are selected from the second input parameters.

5. The method according to claim 1, characterized in that, The method further includes: Based on the transparent fixed value model corresponding to the target data acquisition node, obtain the predefined fixed value parameters; The step of obtaining target output parameters from the data source corresponding to the target data acquisition node based on the target input parameters includes: Based on the target input parameters and the fixed value parameters, the target output parameters are obtained from the data source corresponding to the target data acquisition node.

6. The method according to claim 1, characterized in that, The step of obtaining the target data result from the local parameter pool includes: Retrieve all parameters from the local parameter pool; The results of all the parameters are processed and arranged to obtain the target data results.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: If the execution of the first data acquisition node in the at least one data acquisition node is abnormally interrupted, the abnormal interruption information corresponding to the first data acquisition node shall be output; the abnormal interruption information shall include at least one of the following: abnormal interruption status, abnormal interruption reason, and local result data corresponding to the data acquisition node that was executed before the abnormal interruption.

8. An electronic device, characterized in that, include: A memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the data acquisition method of any one of claims 1 to 7 when the computer program is invoked.

9. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the data acquisition method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the data acquisition method according to any one of claims 1 to 7.

Images