Data processing method and device, equipment, storage medium and program product

Abstract

The invention provides a data processing method and device, equipment, a storage medium and a program product. The data processing method comprises the steps that under the condition that a configuration file of a to-be-tested service meets a data construction condition, a received data processing request is analyzed, interface input parameter information is obtained, and at least one data storage unit identifier corresponding to the to-be-tested service is obtained from the interface input parameter information; determining at least one first data storage unit from a data storage system based on the at least one data storage unit identifier, the data storage system comprising M data storage units storing different object information; and based on the configuration file and the first object information in the first data storage unit, constructing test data of the to-be-tested service. According to the method and the device, the repeatability of the test data generation process can be reduced, and the data processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, equipment, storage medium and program product. Background Art

[0002] Generating test data is also called data creation, which means creating a set of data to test the functions, performance and other characteristics of the business or software system to be tested. These data may be inputs under normal operation conditions or inputs under abnormal or edge conditions. The purpose is to verify whether the behavior and response of the business or software system to be tested in various situations are in line with expectations. Summary of the invention

[0003] The embodiments of the present application provide a data processing method, apparatus, device, storage medium and program product, which can reduce the repeatability of the test data generation process and improve the efficiency of data processing.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a data processing method, the method comprising:

[0006] When the configuration file of the service to be tested meets the data construction condition, the received data processing request is parsed to obtain interface input parameter information, and at least one data storage unit identifier corresponding to the service to be tested is obtained from the interface input parameter information;

[0007] Based on the at least one data storage unit identifier, determining at least one first data storage unit from a data storage system, wherein the data storage system includes M data storage units storing different object information;

[0008] Based on the configuration file and the first object information in the first data storage unit, test data of the service to be tested is constructed.

[0009] The present application provides a data processing device, the device comprising:

[0010] A first acquisition module is used to parse the received data processing request to obtain interface input parameter information when the configuration file of the service to be tested meets the data construction condition, and obtain at least one data storage unit identifier corresponding to the service to be tested from the interface input parameter information;

[0011] A first determining module, configured to determine at least one first data storage unit from a data storage system based on the at least one data storage unit identifier, wherein the data storage system includes M data storage units storing different object information;

[0012] A construction module is used to construct test data of the service to be tested based on the configuration file and the first object information in the first data storage unit.

[0013] An embodiment of the present application provides a computer device, including:

[0014] A memory for storing computer executable instructions;

[0015] The processor is used to implement the data processing method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0016] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for causing a processor to execute the data processing method provided in the embodiment of the present application.

[0017] An embodiment of the present application provides a computer program product, including computer executable instructions, which, when executed by a processor, implement the data processing method provided in the embodiment of the present application.

[0018] The embodiments of the present application have the following beneficial effects:

[0019] In an embodiment of the present application, at least one data storage unit identifier is first obtained from the interface input parameter information, and then at least one first data storage unit is determined from the data storage system based on the at least one data storage unit identifier. In this way, the indication of the first data storage unit is realized through the interface input parameter information, which can improve the setting flexibility of the first data storage unit; based on this, in the case where the business to be tested corresponds to a first data storage unit, a first data storage unit for testing the business to be tested can be determined based on a data storage unit identifier; and in the case where the business to be tested corresponds to multiple first data storage units, multiple different first data storage units can be determined based on multiple data storage unit identifiers, that is, all first data storage units for testing the business to be tested are determined at one time, thereby achieving the purpose of quickly and accurately determining the first data storage unit. Among them, a data storage unit can be considered as a data source scenario. Then, based on the configuration file and the first object information in the first data storage unit, the test data of the business to be tested is constructed. In this way, no matter how many first data storage units correspond to the business to be tested, all the first data storage units used to test the business to be tested can be determined based on the interface input parameter information, and the test data can be constructed at one time based on the configuration file and the first object information in the first data storage unit; that is, the test data can be obtained by executing the data generation or data processing process only once, without executing the data generation or data processing process for each first data storage unit separately, thereby realizing the construction or generation of test data based on multiple first data storage units, that is, realizing the construction of test data based on multiple data source scenarios, which can reduce the execution of data generation or data processing processes, reduce the repeatability of the test data generation process, and improve the efficiency of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is an implementation framework diagram of data processing provided by relevant technologies;

[0021] Figure 2 is a schematic diagram of a network architecture of a data processing system 100 provided in an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of the structure of the server 400 provided in an embodiment of the present application;

[0023] Figure 4 It is a first flow chart of the data processing method provided in the embodiment of the present application;

[0024] Figure 5 It is a schematic diagram of a flow chart of constructing test data provided by an embodiment of the present application;

[0025] Figure 6is a second flow chart of the data processing method provided in an embodiment of the present application;

[0026] Figure 7 It is an implementation framework diagram of the data processing provided in the embodiment of the present application.

[0027] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0029] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0030] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0031] In the embodiments of the present application, the term "module" or "unit" refers to a computer executable instruction or a part of a computer executable instruction with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0032] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0033] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0034] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0035] 1) Extensible Markup Language (XML) is a subset of Standard Generalized Markup Language (SGML) and is used to transmit and store data. The main features of XML include self-descriptiveness, extensibility, and strict syntax requirements. XML tags are customizable, allowing XML to adapt to the marking needs of various data types and structures.

[0036] The role of XML is very wide, mainly including data transmission and storage, structured data, promoting information exchange, improving the accuracy of search and indexing, supporting the transmission of software components, standardization and language conversion, etc.

[0037] 2) JavaScript Object Notation (JSON) is a lightweight data exchange format. JSON is based on JavaScript objects and arrays and uses text to represent complex data structures. JSON is easy for people to read and write, and it is also easy for machines to parse and generate.

[0038] The key features of JSON include key-value pairs, objects and arrays, string representation, strict format, etc.

[0039] 3) Hadoop Base, referred to as hbase, is a distributed and scalable big data storage system. Hbase runs on the Hadoop ecosystem, uses Hadoop's file storage system (Hadoop Distributed FileSystem, HDFS) to store data, and can work with Hadoop's computing framework MapReduce.

[0040] The main features of hbase include distribution and scalability, columnar storage, flexible standards or conventions (schema), high availability and fault tolerance, real-time reading and writing, etc.

[0041] 4) Kafka is an open source stream processing platform developed by LinkedIn and contributed to the Apache Software Foundation. It is mainly used to build real-time data pipelines and stream processing applications. Kafka is designed to provide a way to efficiently process and transmit data in a streaming data storage system.

[0042] 5) Pre-loan approval strategy is an important part of financial risk control and the basis of the entire loan process. It is used to screen target service customers. Building a high-quality pre-loan approval strategy is an important prerequisite for preventing risks and reducing bad debts.

[0043] In order to better understand the data processing method provided in the embodiments of the present application, the data processing method in the related art and its existing shortcomings are first described.

[0044] In the related art, if the required test data comes from different types of data, it is necessary to generate data for each type of data respectively, that is, it is necessary to generate test data for each type of data respectively.

[0045] Figure 1 This is a framework diagram for implementing data processing provided by related technologies, see Figure 1 , Figure 1 For example, the test data comes from four different types of data storage units. Here, the type of data storage unit can be called a scenario, that is, Figure 1 Take the test data from four different scenarios as an example. Figure 1 It includes process 01 of generating test data for scenario 1, process 02 of generating test data for scenario 2, process 03 of generating test data for scenario 3, and process 04 of generating test data for scenario 4. Among them, the type of data storage unit corresponding to scenario 1 is hbase, the type of data storage unit corresponding to scenario 2 is kafka, the type of data storage unit corresponding to scenario 3 is msql, and the type of data storage unit corresponding to scenario 4 is ES.

[0046] Continue to see Figure 1 On the one hand, the process of generating test data in the four scenarios is basically the same, except that the inserted data storage units are different, which makes the implementation process redundant and the execution process repeated; on the other hand, in the process of generating test data, it is necessary to execute four generation processes separately, which increases the number of single executions and brings the problem of generating more dirty data. Based on this, with the increase of data storage units, the above problems become more obvious, that is, the relevant technology will have problems such as low generation efficiency, occupying more computing resources, and generating more dirty data.

[0047] Combining the above-mentioned related technologies, it can be seen that the disadvantages of the related technologies are, first, the configuration file repetition rate is high and the implementation process is redundant, which brings the problem of inefficient test data generation; second, the number of single executions is large, which generates more dirty data; third, the essence of the related technology is to generate test data based on a single type of data, and the related technology cannot generate test data based on multiple types of data.

[0048] The embodiments of the present application provide a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product, which can reduce the repeatability of the test data generation process and improve the efficiency of data processing. The following describes an exemplary application of the computer equipment provided by the embodiments of the present application. The computer equipment provided by the embodiments of the present application can be implemented as a server, and can be implemented as various types of terminals such as a laptop computer, a tablet computer, a desktop computer, a set-top box, a smart phone, a smart speaker, a smart watch, a smart TV, and a vehicle-mounted terminal. Below, an exemplary application when the computer equipment is implemented as a server will be described.

[0049] See also Figure 2 , Figure 2 It is a network architecture diagram of the data processing system 100 provided in an embodiment of the present application. To support a data processing application, the server 400 is connected to the terminal 200 via the network 300, wherein the network 300 may be a wide area network or a local area network, or a combination of the two.

[0050] Continue to see Figure 2 , the terminal 200 generates a data processing request in response to the received data processing instruction, and sends the data processing request to the server 400; the server 400 is used to parse the received data processing request when the configuration file of the service to be tested meets the data construction condition, obtain the interface input parameter information, and obtain at least one data storage unit identifier corresponding to the service to be tested from the interface input parameter information; based on the at least one data storage unit identifier, determine at least one first data storage unit from the data storage system, the data storage system includes M data storage units storing different object information; based on the configuration file and the first object information in the first data storage unit, construct the test data of the service to be tested. The server 400 is also used to return the test data to the terminal 200 so that the terminal 200 can display the test data.

[0051] In the embodiment of the present application, the server 400 first obtains at least one data storage unit identifier from the interface input parameter information, and then determines at least one first data storage unit from the data storage system based on the at least one data storage unit identifier, so that the indication of the first data storage unit is realized through the interface input parameter information, which can improve the setting flexibility of the first data storage unit; based on this, in the case where the business to be tested corresponds to a first data storage unit, a first data storage unit for testing the business to be tested can be determined based on a data storage unit identifier; and in the case where the business to be tested corresponds to multiple first data storage units, multiple different first data storage units can be determined based on multiple data storage unit identifiers, that is, all first data storage units for testing the business to be tested are determined at one time, so as to achieve the purpose of quickly and accurately determining the first data storage unit, wherein a data storage unit can be considered as a data source scenario. Then, based on the configuration file and the first object information in the first data storage unit, the test data of the business to be tested is constructed. In this way, no matter how many first data storage units correspond to the business to be tested, all the first data storage units used to test the business to be tested can be determined based on the interface input parameter information, and the test data can be constructed at one time based on the configuration file and the first object information in the first data storage unit; that is, the test data can be obtained by executing the data generation or data processing process only once, without executing the data generation or data processing process for each first data storage unit separately, thereby realizing the construction or generation of test data based on multiple first data storage units, that is, realizing the construction of test data based on multiple data source scenarios, which can reduce the execution of data generation or data processing processes, reduce the repeatability of the test data generation process, and improve the efficiency of data processing.

[0052] In some embodiments, the server 400 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The video server and the server may be directly or indirectly connected via wired or wireless communications.

[0053] See also Figure 3 , Figure 3 is a schematic diagram of the structure of the server 400 provided in an embodiment of the present application, Figure 3The server 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the server 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 3 Various buses are labeled as bus system 440 .

[0054] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0055] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0056] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0057] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0058] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0059] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0060] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0061] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0062] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0063] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 3 The data processing device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., and includes the following software modules: a first acquisition module 4551, a first determination module 4552 and a construction module 4553. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0064] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (Application Specific Integ rated Circuit, ASIC), DSP, programmable logic device (Programmable Logic Device, P LD), complex programmable logic device (Complex Programmable Logic Device, CPLD), field programmable gate array (Field-Programmable Gate Array, FPGA) or other electronic components.

[0065] In some embodiments, the server or terminal can implement the data processing method provided by the embodiment of the present application by running various computer executable instructions. For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as data generation APP, data testing APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0066] The data processing method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0067] The following describes the data processing method provided by the embodiment of the present application. As mentioned above, the computer device implementing the data processing method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution subject of each step will not be repeatedly described below.

[0068] It should be noted that in the example of the data processing method below, the strategy is described by taking the business to be tested as an example. Those skilled in the art, based on their understanding of the following, can apply the data processing method provided in the embodiment of the present application to the processing of other types of businesses.

[0069] See also Figure 4 , Figure 4 is a first flow chart of the data processing method provided in the embodiment of the present application, which will be combined with Figure 4 The steps shown are used to illustrate the data processing method provided in the embodiment of the present application. Figure 4 The execution subject is described by taking a server as an example.

[0070] In step S101, when the configuration file of the service to be tested meets the data construction condition, the received data processing request is parsed to obtain interface input parameter information, and at least one data storage unit identifier corresponding to the service to be tested is obtained from the interface input parameter information.

[0071] In the embodiment of the present application, the business to be tested may be a risk control approval strategy, an information verification strategy, an abnormality determination strategy, etc.

[0072] In some embodiments, whether the configuration file meets the data construction condition can be determined based on whether the configuration file is obtained. Based on this, if the configuration file can be successfully obtained, it is considered that the configuration file meets the data construction condition; if the configuration file cannot be obtained, it is considered that the configuration file does not meet the data construction condition.

[0073] In some embodiments, it is also possible to determine whether the configuration file meets the data construction condition based on whether the configuration file can be obtained and whether the configuration file can be successfully verified. Based on this, if the configuration file can be successfully obtained and the configuration file is verified, it is considered that the configuration file meets the data construction condition, wherein the verification of the configuration file may include at least one of syntax verification, structure verification, data type verification, and content verification; and if the configuration file cannot be obtained or the configuration file verification fails, it is considered that the configuration file does not meet the data construction condition.

[0074] In some embodiments, the data processing request may be a data construction request, which is used to request data construction and may also be considered to be used to start a data construction process. When the configuration file of the service to be tested meets the data construction conditions, the serialization format of the data may be determined according to the request header in the data processing request, and then the parser corresponding to the serialization format may be used to perform a parsing operation to obtain the interface input parameter information.

[0075] In some embodiments, the interface input parameter information includes at least one data storage unit identifier, based on which, at least one data storage unit identifier can be obtained from the interface input parameter information. When a data processing request is received, on the one hand, the number generation interface is automatically called, and on the other hand, the data processing request is also parsed to obtain the interface input parameter information, which can be set in advance. Since at least one data storage unit identifier can be obtained from the interface input parameter information, the interface input parameter information can indicate the first data storage unit storing the test data.

[0076] In some embodiments, taking the format of the data processing request as JSON for example, the data processing request can be parsed by a JSON parser to obtain interface input parameter information.

[0077] In some embodiments, before executing the above step S101, the implementation method of determining whether the business to be tested meets the construction conditions can be: when the configuration file is obtained, the format of the configuration file is checked to obtain a verification result; when the verification result is that the verification passes, it is determined that the configuration file of the business to be tested meets the data construction conditions.

[0078] Taking the form of a configuration file for extensible markup language as an example, format verification includes syntax verification, structure verification, data type verification, content verification, namespace verification, entity reference verification, comment and processing instruction verification, etc.

[0079] Among them, syntax checking refers to checking whether the XML document conforms to XML syntax rules, including the correct use of elements, attributes, character references, etc.

[0080] Structural validation ensures that the structure of an XML document complies with the rules defined by DTD or XML Schema, including the existence, order, and nesting of elements and attributes. Data type validation includes verifying the correctness of data types in XML documents, such as strings, numbers, dates, etc.

[0081] Content validation refers to checking whether the content in an XML document conforms to a specific content model, including the length of a string, the value of an enumeration, etc.

[0082] Namespace validation is to ensure that the namespaces used in the XML document are valid and correctly declared and referenced.

[0083] Entity reference verification refers to checking whether entities are correctly referenced and declared in XML documents, such as character references, entity references, etc.

[0084] Comment and processing instruction validation is to ensure that the use of comments and processing instructions complies with the rules, thereby facilitating successful parsing of comments and processing instructions.

[0085] In some embodiments, if the configuration file cannot be obtained, the process is terminated directly. If the verification result is that the verification fails, it is determined that the configuration file of the service to be tested does not meet the data construction condition.

[0086] In some embodiments, before receiving a data processing request, a configuration file needs to be created first, that is, before executing the above step S101, the configuration file can also be created through the following steps S001A to S004A, which are described in detail below.

[0087] In step S001A, based on the service identifier of the service to be tested, a first data storage structure and a parameter identifier set are determined.

[0088] In some embodiments, a preset mapping table may be constructed in advance, which stores a service identifier, a data storage structure, a parameter identifier, a first mapping relationship between the service identifier and the data storage structure, and a second mapping relationship between the service identifier and the parameter identifier.

[0089] The first data storage structure may refer to a data storage structure that may need to be called during the process of testing the service to be tested. It can also be considered that the first data storage structure is used to store test data, wherein the test data is used to test the service to be tested. The first data storage structure is a structure set; the first data storage structure includes the data storage structure required for testing the service to be tested, and the first data storage structure may also include the data storage structure required for testing other test services, that is, the data storage structure required for testing the service to be tested is included in the first data storage structure.

[0090] In some embodiments, the parameter identification set may refer to parameter identifications of all test data, where the test data may be considered as parameter values, wherein the test data refers to data used to test the business to be tested. Exemplarily, the parameter identification set may be (apply_no, cooperator_id, TYPEL, trade_type, trade_amt, manual_check, rule_result).

[0091] In some embodiments, the first data storage structure may be determined based on the service identifier of the service to be tested and the first mapping relationship; and the parameter identifier set may be determined based on the service identifier of the service to be tested and the second mapping relationship.

[0092] Exemplarily, assuming that the data storage structures required for testing the business to be tested are data storage structure a, data storage structure b, and data storage structure c, the first data storage structure can be data storage structure a, data storage structure b, and data storage structure c; the first data storage structure can also be data storage structure a, data storage structure b, data storage structure c, and data storage structure d. In this case, data storage structure d can be randomly determined, and data storage structure d can also be the most universal data storage structure. Among them, data storage structure a can be hbase, data storage structure b can be kafka, and data storage structure c can be msql.

[0093] In step S002A, a second data storage unit adopting a first data storage structure is determined from the data storage system.

[0094] In the embodiment of the present application, the second object information and the first parameter identifier of each second object information are stored in the second data storage unit, and the second object information is stored using the first data storage structure. The amount of information included in the second object information is greater than the amount of information included in the first object information. Exemplarily, the second object information may include: object identifier, object name, instant messaging account, gender, age, address, object click event, object login log object account points, object account level, etc. In addition, each information of the second object information has a parameter identifier, which is recorded as the first parameter identifier.

[0095] Here, the data storage unit adopting the first data storage structure may be determined as the second data storage unit.

[0096] In some embodiments, the data storage system includes M data storage units storing different object information, where M is a positive integer. Generally, M is a positive integer greater than 1. For example, the value of M may be 4, 6, 7, etc. Since different data storage units store different object information, and different object information has different representation forms, for example, user names and user login logs have different representation forms, different data storage units use different data storage structures in order to efficiently store and manage different object information. For example, the data storage structures used by the data storage units may be hbase, kafka, mysql, es, etc.

[0097] Continuing with the above example, assume that the first data storage structure is data storage structure a, data storage structure b, data storage structure c and data storage structure d, and data storage unit 1 uses data storage structure a to store data (that is, the second object information), data storage unit 2 uses data storage structure b to store data, data storage unit 3 uses data storage structure c to store data, and data storage unit 4 uses data storage structure d to store data. Based on this, data storage unit 1, data storage unit 2, data storage unit 3 and data storage unit 4 are determined as second data storage units.

[0098] In step S003A, the first parameter identifier stored in both the second data storage unit and the parameter identifier set is determined as the second parameter identifier.

[0099] In some embodiments, for each data storage unit in the second data storage unit, the data storage unit is data stored using the same data storage structure, and the configuration file corresponding to the data storage unit includes multiple parameter identifiers, which can be recorded as first parameter identifiers, and the first parameter identifier in the parameter identifier set is determined as the second parameter identifier. Exemplarily, assuming that the parameter identifier set is (parameter identifier 1, parameter identifier 2, parameter identifier 5, parameter identifier 11, parameter identifier 13, parameter identifier 15, parameter identifier 16), and the first parameter identifier includes parameter identifiers 1 to 18, then parameter identifier 1, parameter identifier 2, parameter identifier 5, parameter identifier 11, parameter identifier 13, parameter identifier 15 and parameter identifier 16 are determined as second parameter identifiers. That is, the first parameter identifier included in the parameter identifier set is determined as the second parameter identifier.

[0100] Taking the data storage structure adopted by the second data storage unit including hbase and kafka as an example, for the data storage unit adopting the hbase data storage structure, the data storage unit may include apply_no, cooperator_id, acct_type and agent_fee_amt first parameter identifiers. When the parameter identifier set is (apply_no, cooperator_id, TYPEL, trade_type, trade_amt, manual_check, rule_result), apply_no and cooperator_id therein may be determined as the second parameter identifier. For the data storage unit adopting the kafka data storage structure, the data storage unit may include TYPEL, RSN_INQ, INSI, TIME_INST, ID_UNQP and TIMEI first parameter identifiers. With reference to the above parameter identifier set, TYPEL therein may be determined as the second parameter identifier. That is, apply_no, cooperator_id and TYPEL may be determined as the second parameter identifier in the end.

[0101] In step S004A, a configuration file is created based on the second parameter identifier.

[0102] In some embodiments, the second data storage unit further stores a topic identifier of the second parameter identifier, and the topic identifier is used to uniquely identify a specific message flow or data channel. Exemplarily, the topic identifier can be apply_main, GAA01_LOAN_L_DETAIL, dbustaf_trade_info, rule_result_info, etc. Taking the GAA01_LOAN_L_DETAIL topic identifier as an example, the topic identifier is used to classify and organize messages related to loans, so that relevant data of loan details can be sent to this topic. The business semantics of the message content can be provided by the topic identifier, so that data exchange and processing can be carried out effectively.

[0103] For each data storage unit in the second data storage unit, the data storage unit and the included second parameter identifier and the subject identifier of the second parameter identifier are modularly packaged to obtain a sub-configuration file of the data storage unit; then, each sub-configuration file is combined to obtain a configuration file for the service to be tested.

[0104] In some embodiments, the structure identifier of the first data storage structure and the preset initial value of the second parameter identifier are also obtained. Based on this, the above-mentioned modular packaging of the second parameter identifier and the subject identifier of the second parameter identifier includes combining the structure identifier, the second parameter identifier, the subject identifier of the second parameter identifier and the preset initial value of the second parameter identifier according to a preset format to obtain the combined information, and then packaging the combined information to obtain a sub-configuration file.

[0105] In some embodiments, a file template of the configuration file may also be obtained, and then the file template may be filled with the structure identifier, the preset initial value of the second parameter identifier, the second parameter identifier in the second data storage unit, and the subject identifier of the second parameter identifier to obtain the configuration file.

[0106] Through the above steps S001A to S004A, the first data storage structure, the parameter identification set and the second data storage unit can be determined first, and then the second parameter identification can be determined based on the second data storage unit, and finally a configuration file can be created. Since the data storage structure for testing the business to be tested is included in the second data storage structure, if the business to be tested changes, and the data storage structure for testing the changed business to be tested is still included in the second data storage structure, the configuration file created based on the second data storage structure can be applied to the changed business to be tested, thereby expanding the scope of application of the configuration file. That is, the configuration file created through the above steps S001A to S004A is a configuration file with a wide range of applications and strong universality, so that test data of different businesses to be tested can be constructed based on the configuration file, simplifying the data processing process and improving the efficiency of data processing.

[0107] In some embodiments, the interface input parameter information includes an input parameter identifier, which may be the name of an input parameter. The input parameter identifier is used to identify or indicate a parameter passed in a function or method. Here, the input parameter identifier is used to indicate a data storage unit identifier. The implementation method of "obtaining at least one data storage unit identifier corresponding to the service to be tested from the interface input parameter information" in the above step S101 may be to first obtain the parameter value of the input parameter identifier from the interface input parameter information; and then determine the parameter value of the input parameter identifier as at least one data storage unit identifier. Exemplarily, the interface input parameter information may be: "make": "h, k", and the input parameter identifier may be "make", then the parameter value of the input parameter identifier is "h, k", based on which, h and k are determined as data storage unit identifiers.

[0108] In this way, at least one data storage unit identifier can be quickly and accurately determined through the parameter value of the input parameter identifier.

[0109] Continue to see Figure 4 , and the description continues with step S101 above.

[0110] In step S102, at least one first data storage unit is determined from the data storage system based on at least one data storage unit identifier.

[0111] In an embodiment of the present application, the data storage system includes M data storage units storing different object information, where M is a positive integer. Generally, M is a positive integer greater than 1. For example, the value of M can be 4, 6, 7, etc.

[0112] In some embodiments, different object information represents information stored using different data storage structures, based on which each data storage unit uses different data storage structures to store object information. The object information may include a public profile of the object, the gender of the object, the age of the object, the industry in which the object is engaged, etc. The object information may also include an object ID, an object name, an email address, an address, an object click event, an object login log, an object purchase log, etc.

[0113] In some embodiments, the above step S102 "determining at least one first data storage unit from the data storage system based on at least one data storage unit identifier" can be implemented by, for each data storage unit identifier, determining the first data storage unit corresponding to the data storage unit identifier from the data storage system.

[0114] In some embodiments, the data storage unit corresponding to the data storage unit identifier is determined as the first data storage unit. Continuing with the above example, if the data storage unit identifier is h, data storage unit 1 is determined as the first data storage unit; if the data storage unit identifiers are h and k, data storage unit 1 and data storage unit 2 are jointly determined as the first data storage unit.

[0115] In step S103, test data of the service to be tested is constructed based on the configuration file and the first object information in the first data storage unit.

[0116] In some embodiments, the fourth object information is first obtained from the first object information of each first data storage unit based on the configuration file, and then each fourth object information is combined to obtain the test data of the service to be tested.

[0117] In some embodiments, the configuration file may be formatted first to obtain a converted configuration file; then, based on the converted configuration file, fourth object information may be obtained from the first object information of each first data storage unit; finally, each fourth object information may be combined to obtain test data for the service to be tested.

[0118] In some embodiments, see Figure 5 The above step S103 can be implemented by the following steps S1031 to S1035, which are described in detail below.

[0119] In step S1031, the second parameter identifier stored in the configuration file and the third parameter identifier stored in the first data storage unit are obtained.

[0120] In the embodiment of the present application, the field identifier included in the configuration file is determined as the second parameter identifier, and the field identifier included in each first data storage unit is determined as the third parameter identifier. Exemplarily, the second parameter identifier may include parameter identifier 1, parameter identifier 2, parameter identifier 5, parameter identifier 11, parameter identifier 13, parameter identifier 15 and parameter identifier 16, and the third parameter identifier may include parameter identifiers 1 to 10.

[0121] In some embodiments, during the data processing process, part of the first object information in each first data storage unit is needed, but not all of the first object information in the first data storage unit is needed.

[0122] In step S1032, the third parameter identifier stored in the first data storage unit and not stored in the configuration file is determined as the fourth parameter identifier, and a preset parameter value of the fourth parameter identifier is obtained.

[0123] In some embodiments, the parameter identifiers other than the second parameter identifier in the third parameter identifier are determined as the fourth parameter identifier. Continuing with the above example, if the second parameter identifier may include parameter identifier 1, parameter identifier 2, parameter identifier 5, parameter identifier 11, parameter identifier 13, parameter identifier 15, and parameter identifier 16, and the third parameter identifier may include parameter identifiers 1 to 10, then parameter identifier 3, parameter identifier 4, parameter identifier 6, parameter identifier 7, parameter identifier 8, parameter identifier 9, and parameter identifier 10 are determined as the fourth parameter identifier.

[0124] In actual implementation, the second parameter identifier and the third parameter identifier can be considered as two identifier sets. Based on this, the intersection of the second parameter identifier and the third parameter identifier can be determined first, recorded as the sixth parameter identifier, and then the difference between the third parameter identifier and the sixth parameter identifier can be determined. Finally, the difference set is determined as the fourth parameter identifier, and the fourth parameter identifier can also be considered as an identifier set.

[0125] In the embodiment of the present application, the preset parameter value is information set in advance. When performing data processing, the parameter value of the second parameter is required, and the parameter value of the fourth parameter is not required. Based on this, each preset parameter value to be identified by the fourth parameter can be set to "null".

[0126] In step S1033, the configuration file is converted into a format to obtain a converted configuration file that supports data insertion.

[0127] In some embodiments, the process of format conversion may include: determining a target format, selecting a conversion strategy, and converting according to the conversion strategy. The target format is a format determined according to actual needs, and here, the target format refers to a format that supports data insertion; selecting a conversion strategy refers to determining a conversion strategy for converting a configuration file from a current format to a target format from among the conversion strategies to be selected; converting according to the conversion strategy refers to converting the configuration file from a current format to a target format using the conversion strategy to obtain a converted configuration file.

[0128] Exemplarily, since the configuration file in XML format does not support data insertion, while the configuration file in JSON format supports data insertion, the configuration file in XML format may be converted into a configuration file in JSON format.

[0129] In step S1034, the fourth parameter identifier, the preset parameter value, and the first object information of the second parameter identifier are added to the converted configuration file to obtain a processed configuration file.

[0130] In some embodiments, the preset parameter value is determined as the parameter value of the fourth parameter. Since the fourth parameter identifier is not stored in the configuration file, and the parameter identifier stored in the converted configuration file is consistent with the parameter identifier stored in the configuration file, that is, the fourth parameter identifier is not stored in the converted configuration file. Based on this, the fourth parameter identifier and the preset parameter value are added to the converted configuration file, that is, the fourth parameter identifier and the parameter value corresponding to the fourth parameter identifier are added to the converted configuration file, thereby obtaining a processed configuration file with a complete parameter identifier. In addition, the first object information of the second parameter identifier is also added to the converted configuration file, so as to update the preset initial value of the second parameter identifier using the first object information of the second parameter identifier, thereby obtaining a processed configuration file containing the first object information. Combining the above two adding processes, a processed configuration file with a complete parameter identifier and containing object information can be finally obtained.

[0131] In step S1035, test data of the service to be tested is constructed based on the processed configuration file.

[0132] In some embodiments, during the process of loading the processed configuration file, the third object information of the fifth parameter identifier is called from the first data storage unit included in the processed configuration file, and the third object information is determined as the test data. The fifth parameter identifier refers to a parameter identifier included in both the first data storage unit and the configuration file, and the fifth parameter identifier refers to a parameter identifier used to test the service to be tested.

[0133] In some embodiments, the above step S1035 can be implemented by following the steps S0351 to S0352, which are described in detail below.

[0134] In step S0351, the third parameter identifier stored in both the first data storage unit and the configuration file is determined as the fifth parameter identifier.

[0135] In some embodiments, since a third parameter identifier is stored in the first data storage unit and a second parameter identifier is stored in the configuration file, the second parameter identifier and the third parameter identifier can be considered as an identifier set. Therefore, the parameter identifier included in the second parameter identifier and the third parameter identifier can be determined as the fifth parameter identifier.

[0136] Continuing with the above example, if the second parameter identifier can include parameter identifier 1, parameter identifier 2, parameter identifier 5, parameter identifier 11, parameter identifier 13, parameter identifier 15 and parameter identifier 16, and the third parameter identifier can include parameter identifier 1 to parameter identifier 10, then parameter identifier 1, parameter identifier 2 and parameter identifier 5 are determined as the fifth parameter identifier.

[0137] The second parameter identifier in each first data storage unit is determined as the sixth parameter identifier of the first data storage unit, that is, the sixth parameter identifier is screened and determined from the third parameter identifier of each first data storage unit.

[0138] In step S0352, the third object information identified by the fifth parameter in the processed configuration file is determined as test data of the service to be tested.

[0139] In some embodiments, since the processed configuration file stores the first object information corresponding to the third parameter identifier, and the fifth parameter identifier is included in the third parameter identifier, the third object information of the fifth parameter identifier can be determined from the first object information based on the fifth parameter identifier, and the third object information can be determined as the test data for the business to be tested.

[0140] Through the above steps S0351 to S0352, the test data of the service to be tested can be obtained at one time from the first object information stored in the processed configuration file based on the fifth parameter identifier.

[0141] Through the above steps S1031 to S1035, on the one hand, the configuration file will be converted into a converted configuration file that supports data insertion, and then the fourth parameter identifier, the preset parameter value and the first object information will be added to the converted configuration file to obtain a processed configuration file with complete parameter identifiers and containing object information; based on this, the processed configuration file can be successfully loaded; on the other hand, based on the fifth parameter identifier, the test data of the business to be tested can be obtained at one time from the first object information stored in the processed configuration file, thereby improving the efficiency of data processing.

[0142] Through the above steps S101 to S103, at least one data storage unit identifier is first obtained from the interface input parameter information, and then at least one first data storage unit is determined from the data storage system based on the at least one data storage unit identifier. In this way, the first data storage unit is indicated through the interface input parameter information, which can improve the setting flexibility of the first data storage unit; based on this, when the business to be tested corresponds to a first data storage unit, a first data storage unit for testing the business to be tested can be determined based on a data storage unit identifier; and when the business to be tested corresponds to multiple first data storage units, multiple different first data storage units can be determined based on multiple data storage unit identifiers, that is, all first data storage units for testing the business to be tested are determined at one time, so as to achieve the purpose of quickly and accurately determining the first data storage unit, wherein a data storage unit can be considered as a data source scenario. Then, based on the configuration file and the first object information in the first data storage unit, the test data of the business to be tested is constructed. In this way, no matter how many first data storage units correspond to the business to be tested, all the first data storage units used to test the business to be tested can be determined based on the interface input parameter information, and the test data can be constructed at one time based on the configuration file and the first object information in the first data storage unit; that is, only one data generation or data processing process needs to be performed to obtain the test data, without the need to perform data generation or data processing for each first data storage unit, thereby realizing the construction or generation of test data based on multiple first data storage units, that is, realizing the construction of test data based on multiple data source scenarios, which can reduce the execution of data generation or data processing processes, and can also reduce the repeatability of the test data generation process and improve the efficiency of data processing. In addition, the configuration file will be converted into a converted configuration file that supports data insertion as the basis for generating a processed configuration file with complete parameter identification and containing object information, based on which it is ensured that the processed configuration file can be successfully loaded.

[0143] In some embodiments, see Figure 6 After the above step S103, the following steps S104 to S106 may also be performed, which are described in detail below.

[0144] In step S104, a preset strategy and a reference value of the service to be tested are obtained.

[0145] In the embodiment of the present application, the preset strategy can be a risk control approval strategy, an information verification strategy, an abnormality determination strategy, etc., that is, the preset strategy is the business to be tested. The reference value is a value set in advance, and for example, the reference value can be 10, 12, 15, etc.

[0146] In step S105, a logical operation is performed on the test data based on a preset strategy to obtain a test value of the service to be tested.

[0147] Exemplarily, it is assumed that the preset strategy is to perform logical AND operations on two test data in sequence to obtain multiple operation results; and then perform logical OR operations on the multiple operation results to obtain the test value of the service to be tested.

[0148] In step S106, the test result of the service to be tested is determined based on the reference value and the test value.

[0149] In the embodiment of the present application, when the reference value is equal to the test value, the test result is determined to be a test success; and when the test value is not equal to the reference value, the test result is determined to be a test failure.

[0150] In some embodiments, when the test result is a test failure, the preset strategy is adjusted to obtain an adjusted strategy, and then the above steps S105 and S106 are performed based on the adjusted strategy until the test result is a test success, thereby obtaining an adjusted strategy for a successful test. Based on this, the adjusted strategy can be applied to actual application scenarios. For example, the adjusted risk control approval strategy is used to perform automated risk control approval, the information verification strategy is used to achieve automated verification of information, and the anomaly determination strategy is used to achieve anomaly detection.

[0151] Through the above steps S104 to S106, the service to be tested can be tested based on the preset strategy, the reference value of the service to be tested and the test data, and the accuracy and efficiency of the test can be improved through the diversity of the test data. In addition, the availability of the preset strategy can be measured, and when the test result is a test failure, the adjusted strategy can be obtained through adjustment operations, so as to complete the corresponding task using the adjusted strategy.

[0152] The following is an explanation of an exemplary application of an embodiment of the present application in a practical application scenario.

[0153] The embodiment of the present application proposes a data processing method. When data processing is performed using the data processing method provided by the embodiment of the present application, a configuration file is first created. The format of the configuration file may be an Extensible Markup Language (XML) format.

[0154] In some embodiments, the configuration file may define field information of different types of data tables or message bodies. In the configuration file, <test>< / test> Represents the use case of number creation; <variablename>< / variablename> Indicates the name of the field to be compared, for example: <variablename> wkxtAllDebt< / variablename> Indicates that the field to be compared is the wkxtAllDebt field;<type>< / type> Indicates the type of field to be compared, for example: <type> int< / type> Indicates that the type of the wkxtAllDebt field to be compared is an integer type; <preresult>< / preresult> Indicates the expected result of the field to be compared, for example: <preresult> 10< / preresult> Indicates that the expected result of the wkxtAllDebt field to be compared is 10.

[0155] In some embodiments, the configuration file <casedatas>< / casedatas> Represents the data preparation of the scenario table for the use case test;<hbase.apply_main>< / hbase.apply_main> Represents the application_main table of the HBase scenario; for example,<apply_no> 111< / apply_no> Indicates that the apply_no field value in the apply_main table of the hbase scenario is 111.<cooperator_id> 000111< / cooperator_id> Indicates that the value of the cooperator_id field in the apply_main table of the hbase scenario is 000111.

[0156] In some embodiments, the configuration file<kafka.GAA01_LOAN_L_DETAIL>< / kafka.GAA01_LOAN_L_DETAIL> Represents the GAA01_LOAN_L_DETAIL message body of the Kafka scenario; for example, <typel> Personal purchase history< / typel> Indicates that the value of the TYPEL field in the GAA01_LOAN_L_DETAIL message body in the Kafka scenario is a personal purchase record.

[0157] In some embodiments, the configuration file<msql.dbustaf_trade_info>< / msql.dbustaf_trade_info> The dbustaf_trade_info table represents the msql scenario; for example,<trade_type> Auth< / trad e_type> Indicates that the trade_type field value in the dbustaf_trade_info table in the msql scenario is Auth, 99< / trade_amt> Indicates that the value of the trade_amt field in the dbustaf_trade_info table in the msql scenario is 99.

[0158] In some embodiments, the configuration file<es.rule_result_info>< / es.rule_result_info> Represents the rule_result_info table of the es scenario; for example,<manual_check> N< / manual_check> Indicates that the manual_check field value in the rule_result_info table of the es scenario is N.<rule_result> 99< / rule_result> Indicates that the value of the rule_result field in the rule_result_info table in the es scenario is 99.

[0159] In some embodiments, the data processing method includes the following steps 1, 2 and 3, which are described in detail below.

[0160] Step 1: Configure the .xml file according to the test scenario to obtain the xml configuration file.

[0161] Step 2: When data needs to be generated during the test, call the data generation interface.

[0162] Step 3: Pull the XML configuration file during the interface call.

[0163] In some embodiments, the above step three can be implemented by following the steps S701 to S705, which are described in detail below.

[0164] In step S701, it is determined whether the XML configuration file exists.

[0165] In some embodiments, if the XML configuration file can be successfully pulled, it is determined that the XML configuration file exists, and the process proceeds to step S702; if the XML configuration file fails to be pulled, it is determined that the XML configuration file does not exist, and the process proceeds to step S705.

[0166] In step S702, the format of the XML configuration file is checked to obtain a format check result.

[0167] Here, when it is determined that the XML configuration file exists, the format of the XML configuration file is checked to obtain a format check result.

[0168] In some embodiments, the format verification of the XML configuration file can be implemented in the following three ways:

[0169] Method 1: Use an XML validator. You can use open source and commercial tools to validate XML configuration files. For example, you can use tools such as XMLSpy or IBM's XML Developer's Kit. These tools usually provide a graphical user interface that can instantly check syntax and format errors when writing XML configuration files.

[0170] Method 2. Write a script or use a programming language. You can write a script to check the format of the XML configuration file. There are many programming languages ​​that can process XML, such as Python, Java, C#, etc. These languages ​​usually have libraries or modules that can be used to parse and validate XML files.

[0171] Method 3: Use regular expressions. If you need to quickly check the XML configuration file, you can use regular expressions to match and verify the XML format, which can quickly and conveniently implement the check.

[0172] In some embodiments, for verifying the format of the XML configuration file, the verification scope includes whether the start and end of the tag match, whether the tags are nested correctly, whether the attribute values ​​conform to the expected format and data type, whether there are extra whitespace characters or unclosed tags, whether the XML naming rules are followed, etc. Exemplarily, the naming rules can be whether the tag names are all lowercase letters.

[0173] In step S703, it is determined whether the format check result indicates that the check has passed.

[0174] In some embodiments, if the format verification result indicates that the verification has passed, that is, the format of the XML configuration file is correct, then the process proceeds to step S704; if the format verification result indicates that the verification has not passed, that is, the format of the XML configuration file is incorrect, then the process proceeds to step S705.

[0175] In step S704, the format of the xml configuration file is converted into json format, and the format is generated according to the value of the make field of the interface input parameter.

[0176] In an embodiment of the present application, the format of the xml configuration file is converted into the json format to obtain a converted configuration file, which is an insertable file, that is, the converted configuration file supports inserting at least one of the hbase table, kafka message body, msql table and es table into itself.

[0177] In some embodiments, there are four single scenarios:

[0178] In single scenario 1, when the parameter "make": "h" is entered, the data converted to JSON format is processed and then inserted into the HBase table to obtain the inserted file. Then, the test data is obtained from the inserted file, and the actual calculation result is obtained through calculation. The actual calculation result is compared with the expected result to obtain the comparison result, and the process ends.

[0179] In some embodiments, processing the data converted into JSON format means that, since the XML configuration file only includes some fields in the HBase table, but not all fields in the HBase table, it is also necessary to add the fields not included in the XML configuration file to the data converted into JSON format, that is, to add the fields not included in the XML configuration file to the converted configuration file, and assign the added fields to "null" to obtain the processed file.

[0180] In some embodiments, inserting into the hbase table means inserting the hbase table into the processed file, that is, updating the values ​​of non-empty fields in the processed file to corresponding field values ​​in the hbase table to obtain the inserted file.

[0181] In single scenario 2, when the parameter "make":"k" is entered, the data converted to JSON format is processed and then inserted into the Kafka message body to obtain the inserted file. The test data is then obtained from the inserted file, and the actual calculation result is obtained through calculation. The actual calculation result is compared with the expected result to obtain the comparison result, and the process ends.

[0182] In single scenario three, when the parameter "make":"m" is entered, the data converted to json format is processed and then inserted into the msql table to obtain the inserted file. The test is then obtained from the inserted file, and the actual calculation result is obtained through calculation. The actual calculation result is compared with the expected result to obtain the comparison result, and the process ends.

[0183] In single scenario 4, when the parameter "make": "e" is entered, the data converted to JSON format is processed and then inserted into the es table to obtain the inserted file. The test data is then obtained from the inserted file, and the actual calculation result is obtained through calculation. The actual calculation result is compared with the expected result to obtain the comparison result, and the process ends.

[0184] In some embodiments, there are the following nine multi-scenario combinations:

[0185] In the first scenario, when the parameter "make":"h,k" is entered, the data converted to JSON format is processed and then inserted into the HBase table and the Kafka message body to obtain the inserted file. The test data is then obtained from the inserted file and the actual calculation result is obtained through calculation. The actual calculation result is compared with the expected result to obtain the comparison result, and the process ends.

[0186] In the second multi-scenario combination, when the parameter "make":"h,m" is entered, the data converted to json format is processed and then inserted into the hbase table and the msql table to obtain the inserted file. The test data is then obtained from the inserted file, and the actual calculation result is obtained through calculation. The actual calculation result is compared with the expected result to obtain the comparison result, and the process ends.

[0187] In multi-scenario combination three, when the parameter "make":"h,e" is entered, the data converted to json format is processed and then inserted into the hbase table and es table to obtain the inserted file. The test data is then obtained from the inserted file, and the actual calculation result is obtained through calculation. The actual calculation result is compared with the expected result to obtain the comparison result, and the process ends.

[0188] In multi-scenario combination 4, when the parameter "make":"k,m" is entered, the data converted to json format is processed, and then inserted into the kafka message body and msql table to obtain the inserted file. Then, the test data is obtained from the inserted file, and the actual calculation result is obtained through calculation. The actual calculation result is compared with the expected result to obtain the comparison result, and the process ends.

[0189] In multi-scenario combination 5, when the parameter "make":"k,e" is entered, the data converted to JSON format is processed, and then inserted into the Kafka message body and es table to obtain the inserted file. Then, the test data is obtained from the inserted file, and the actual calculation result is obtained through calculation. The actual calculation result is compared with the expected result to obtain the comparison result, and the process ends.

[0190] Multi-scenario combination six: when the parameter "make":"h,k,m" is entered, the data converted to json format is processed and then inserted into the hbase table, kafka message body and msql table to obtain the inserted file. Then the test data is obtained from the inserted file, and the actual calculation result is obtained through calculation. The actual calculation result is compared with the expected result to obtain the comparison result, and the process ends.

[0191] In multi-scenario combination seven, when the parameter "make":"h,k,e" is entered, the data converted to json format is processed and then inserted into the hbase table, kafka message body and es table to obtain the inserted file. Then the test data is obtained from the inserted file, and the actual calculation result is obtained through calculation. The actual calculation result is compared with the expected result to obtain the comparison result, and the process ends.

[0192] Multi-scenario combination eight, when the parameter "make":"h,m,e" is entered, the data converted to json format is processed, and then inserted into the hbase table, msql table and es table to obtain the inserted file, and then the test data is obtained from the inserted file, and the actual calculation result is obtained through calculation. The actual calculation result is compared with the expected result to obtain the comparison result, and the process ends.

[0193] Multi-scenario combination nine, when the parameter "make":"h,k,m,e" is entered, the data converted to json format is processed and then inserted into the hbase table, kafka message body, msql table and es table. Then the program obtains the data for calculation, the actual calculation result is compared with the expected result, and the program ends.

[0194] In step S705, the process ends.

[0195] Here, if the XML configuration file cannot be obtained or the format verification of the XML configuration file is incorrect, the program ends, that is, the process ends.

[0196] The key step of the embodiment of the present application is to design the input parameter make from calling test data for a single scenario to calling test data for a combination of multiple scenarios. The design based on the input parameter make value realizes the data creation from a single scenario to a combination of multiple scenarios, thereby improving the utilization rate of configuration files and codes. And for the situation of scenario combination, the number of calls is reduced. Among them, the configuration of the configuration file and the design of the input parameter value can be combined with different scenarios according to business needs, so as to create test data based on data in different scenarios, and finally compare the results, which greatly improves the utilization rate of test data and reduces the generation of dirty data.

[0197] Figure 7 This is a framework diagram for implementing data processing provided in the embodiment of the present application, see Figure 7 , the implementation framework includes a calling component 801, an acquisition component 802, a judgment component 803, a conversion component 804, an insertion component 805, a calculation component 806 and a comparison component 807. Among them, the calling component 801 is used to call the interface; the acquisition component 802 is used to obtain the XML configuration file and also used to obtain test data; the judgment component 803 is used to determine whether the XML configuration file can be obtained and whether the format of the XML configuration file is correct; the conversion component 804 is used to convert the format of the XML file into JSON; the insertion component 805 is used to insert a table or message body corresponding to the input parameter; the calculation component 806 is used to perform calculations based on the test data to obtain the test results; the comparison component 807 is used to compare the test results with the expected results and obtain the comparison results.

[0198] In some embodiments, for the risk control and approval system business, each customer needs to make a pre-loan approval strategy judgment, where the pre-loan approval strategy includes six links: personal information verification, access rules, fraud determination, blacklist, credit risk assessment, and credit granting. These six links require different data and data results to support them during the testing process, and their data storage methods are different.

[0199] Through the data processing method provided in the embodiment of the present application, test data for multiple scenarios can be created according to the value of the input parameter make and the xml configuration file, that is, it supports the use of multiple different types of data to construct test data and compare data results, which can greatly improve the utilization rate of data.

[0200] The data processing method provided in the embodiment of the present application can simplify configuration and code, reduce the number of calls and reduce the generation of dirty data. It can also generate test data for multiple scene combinations and compare data results.

[0201] Taking the above-mentioned risk control and approval system business as an example, since each customer needs to make a pre-loan approval strategy judgment, and each customer's information is different, the personal information verification, access rules, fraud judgment, blacklist, credit risk assessment and credit data are also different.

[0202] In some embodiments, the format of the xml configuration file may refer to the file format of the above-mentioned configuration file, and the test data construction for personal information verification, access rules, fraud determination, blacklist, credit risk assessment and credit granting may refer to the above-mentioned steps S701 to S705.

[0203] In some embodiments, taking personal information verification, access rules, and fraud determination as examples, personal information verification corresponds to the above-mentioned multi-scenario combination one, and it is necessary to create test data for hbase and kafka. At this time, the parameter "make":"h,k" is entered to process the data converted into json format, and then inserted into the hbase table and the kafka message body to obtain the inserted file, and then get the test data from the inserted file, and obtain the actual calculation result through calculation, that is, the actual calculation result of the field wkxtAllDebt is obtained, and finally the actual calculation result of the field wkxtAllDebt is compared with the expected result of the field wkxtAllDebt to obtain the comparison result, and the process ends.

[0204] In some embodiments, the access rule corresponds to the above-mentioned multi-scenario combination six, and test data for hbase, kafka, and msql need to be created. At this time, the parameter "make":"h,k,m" is entered to process the data converted into json format, and then inserted into the hbase table, kafka message body, and msql table to obtain the inserted file, and then obtain the test data from the inserted file, and obtain the actual calculation result through calculation, that is, obtain the actual calculation result of the field wkxtAllDeb, and finally compare the actual calculation result of the field wkxtAllDebt with the expected result of the field wkxtAllDebt to obtain the comparison result, and the process ends.

[0205] In some embodiments, fraud determination corresponds to the above-mentioned multi-scenario combination nine, and test data for hbase, kafka, msql, and es need to be created. At this time, the parameter "make":"h,k,m,e" is entered to process the data converted into json format, and then inserted into the hbase table, kafka message body, msql table, and es table to obtain the inserted file, and then obtain the test data from the inserted file, and obtain the actual calculation result through calculation, that is, obtain the actual calculation result of the field wkxtAllDeb, and finally compare the actual calculation result of the field wkxtAllDebt with the expected result of the field wkxtAllDebt to obtain the comparison result, and the process ends.

[0206] The data processing method provided by the embodiment of the present application can simplify configuration and code, reduce the number of calls and dirty data generation. It is also possible to make test data for multiple scenarios according to the input parameter make value and compare the data results. In addition, the purpose of regression test recycling can be achieved through one configuration, reducing manual operations.

[0207] It can be understood that in the embodiments of the present application, data related to the service identifier, the second parameter identifier, the third parameter identifier, the reference value, etc. are involved, and the collection, use and processing of the relevant data need to comply with relevant laws, regulations and standards.

[0208] The following is a description of an exemplary structure of the data processing device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 3 As shown, the software modules stored in the data processing device 455 of the memory 450 may include:

[0209] The first acquisition module 4551 is used to parse the received data processing request to obtain interface input parameter information when the configuration file of the service to be tested meets the data construction condition, and obtain at least one data storage unit identifier corresponding to the service to be tested from the interface input parameter information;

[0210] A first determining module 4552 is used to determine at least one first data storage unit from a data storage system based on the at least one data storage unit identifier, wherein the data storage system includes M data storage units storing different object information;

[0211] The construction module 4553 is used to construct the test data of the service to be tested based on the configuration file and the first object information in the first data storage unit.

[0212] In some embodiments, the software modules in the data processing device 455 may also include:

[0213] A verification module, used to perform format verification on the configuration file when the configuration file is obtained, and obtain a verification result;

[0214] The second determination module is used to determine that the configuration file of the service to be tested meets the data construction condition when the verification result is that the verification is passed.

[0215] In some embodiments, the software modules in the data processing device 455 may also include:

[0216] A third determination module, configured to determine a first data storage structure and a parameter identification set based on a service identification of the service to be tested;

[0217] a fourth determining module, configured to determine, based on the first data storage structure, a second data storage unit from the data storage system, wherein the second data storage unit stores second object information and a first parameter identifier of each second object information, and the data storage structure of the second object information is the first data storage structure;

[0218] A fifth determining module, configured to determine the first parameter identifier in the parameter identifier set in the second data storage unit as the second parameter identifier;

[0219] A creation module is used to create the configuration file based on the second parameter identifier.

[0220] In some embodiments, the construction module 4553 is further used to:

[0221] Obtain a second parameter identifier included in the configuration file, and a third parameter identifier stored in the first data storage unit; determine the third parameter identifier stored in the first data storage unit and not included in the configuration file as a fourth parameter identifier, and obtain a preset parameter value of the fourth parameter identifier; perform format conversion on the configuration file to obtain a converted configuration file that supports data insertion; add the fourth parameter identifier, the preset parameter value, and the first object information of the second parameter identifier to the converted configuration file to obtain a processed configuration file; and construct test data for the business to be tested based on the processed configuration file.

[0222] In some embodiments, the construction module 4553 is further used to:

[0223] The third parameter identifier stored in the first data storage unit and included in the configuration file is determined as the fifth parameter identifier; and the third object information of the fifth parameter identifier in the processed configuration file is determined as the test data of the service to be tested.

[0224] In some embodiments, the first acquisition module 4551 is further used to:

[0225] Acquire the parameter value of the input parameter identifier from the interface input parameter information; and determine the parameter value of the input parameter identifier as the at least one data storage unit identifier.

[0226] In some embodiments, the software modules in the data processing device 455 may also include:

[0227] A second acquisition module, used to acquire a preset strategy and a reference value of the service to be tested;

[0228] A calculation module, used for performing a logical operation on the test data based on the preset strategy to obtain a test value of the service to be tested;

[0229] A sixth determination module is used to determine a test result of the service to be tested based on the reference value and the test value.

[0230] The embodiment of the present application provides a computer program product, which includes computer executable instructions, and the computer executable instructions are stored in a computer readable storage medium. The processor of the computer device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the computer device executes the data processing method described above in the embodiment of the present application.

[0231] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will be caused to execute the data processing method provided by the embodiment of the present application, for example, Figure 4 and Figure 6 The data processing method is shown.

[0232] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0233] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0234] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0235] By way of example, computer executable instructions may be deployed to be executed on one computer device or on multiple computer devices located at one site or on multiple computer devices distributed across multiple sites and interconnected by a communication network.

[0236] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: When the configuration file of the service to be tested meets the data construction condition, the received data processing request is parsed to obtain interface input parameter information, and at least one data storage unit identifier corresponding to the service to be tested is obtained from the interface input parameter information; Based on the at least one data storage unit identifier, determining at least one first data storage unit from a data storage system, wherein the data storage system includes M data storage units storing different object information; Based on the configuration file and the first object information in the first data storage unit, test data of the service to be tested is constructed.

2. The method according to claim 1, characterized in that The method further comprises: When the configuration file is obtained, a format check is performed on the configuration file to obtain a check result; When the verification result is that the verification is passed, it is determined that the configuration file of the service to be tested meets the data construction condition.

3. The method according to claim 1, characterized in that Before receiving the data processing request, the method further includes: Based on the service identifier of the service to be tested, determining a first data storage structure and a parameter identifier set; Determine from the data storage system a second data storage unit using the first data storage structure, wherein the second data storage unit stores second object information and a first parameter identifier of each second object information; Determine the first parameter identifier stored in both the second data storage unit and the parameter identifier set as a second parameter identifier; Based on the second parameter identifier, the configuration file is created.

4. The method according to claim 3, characterized in that The constructing the test data of the service to be tested based on the configuration file and the first object information in the first data storage unit includes: Acquire a second parameter identifier stored in the configuration file and a third parameter identifier stored in the first data storage unit; Determine the third parameter identifier stored in the first data storage unit and not stored in the configuration file as a fourth parameter identifier, and obtain a preset parameter value of the fourth parameter identifier; Performing format conversion on the configuration file to obtain a converted configuration file that supports data insertion; Adding the fourth parameter identifier, the preset parameter value, and the first object information of the second parameter identifier to the converted configuration file to obtain a processed configuration file; The test data of the service to be tested is constructed based on the processed configuration file.

5. The method according to claim 4, characterized in that The constructing the test data of the service to be tested based on the processed configuration file includes: Determine the third parameter identifier stored in both the first data storage unit and the configuration file as a fifth parameter identifier; The third object information identified by the fifth parameter in the processed configuration file is determined as the test data of the service to be tested.

6. The method according to claim 1, characterized in that The interface input parameter information includes an input parameter identifier, and obtaining at least one data storage unit identifier corresponding to the service to be tested from the interface input parameter information includes: Obtain the parameter value of the input parameter identifier from the interface input parameter information; The parameter value of the input parameter identifier is determined as the at least one data storage unit identifier.

7. The method according to any one of claims 1 to 5, characterized in that: After constructing the test data of the service to be tested, the method further includes: Obtaining a preset strategy and a reference value of the service to be tested; Performing a logical operation on the test data based on the preset strategy to obtain a test value of the service to be tested; Based on the reference value and the test value, a test result of the service to be tested is determined.

8. A computer device, characterized in that: The computer device comprises: A memory for storing computer executable instructions; A processor, configured to implement the data processing method according to any one of claims 1 to 7 when executing the computer executable instructions stored in the memory.

9. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program or computer executable instructions, characterized in that: When the computer program or computer executable instructions are executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.

Images