A program testing method and device based on big data order creation

Abstract

The application relates to the technical field of data processing, and provides a program testing method based on big data and related equipment, which comprises the following steps: obtaining a product data configuration table of a first business product, generating first sample data of the first business product based on the product data configuration table, storing the first sample data into a preset relational database, generating a first data form, synchronizing the first data form to a Hive data warehouse, respectively performing business logic calculation on each piece of sample data in the first data form and the second data form through a test program created based on a Spark computing engine and a preset big data algorithm program, obtaining a first calculation result and a second calculation result of each piece of sample data, and determining the correctness of the big data algorithm program in the business logic calculation according to the first calculation result and the second calculation result. The embodiment of the application can test the business logic big data algorithm program of a new business product.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a program testing method and device based on big data. BACKGROUND

[0002] With the increasing business volume of enterprises, the number of new business products of enterprises is gradually increasing every month. Before each new business product goes online, the business logic algorithm of the product needs to be tested. For example, in the life insurance field, different effective premium algorithms are used for each insurance product according to different business types and business properties. Therefore, the effective premium algorithm of the product needs to be tested before the new insurance product goes online, that is, the algorithm program of the effective premium needs to be tested.

[0003] However, there is no inventory data that can be used to perform corresponding program testing for new business products that have not yet gone online. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a program testing method and device based on big data, which can test the business logic algorithm program of a new business product.

[0005] To achieve the above purpose, a first aspect of the embodiments of the present application provides a program testing method based on big data, which comprises:

[0006] Obtaining a product data configuration table of a first business product, and generating first sample data of the first business product based on the product data configuration table;

[0007] Storing the first sample data to a preset relational database to generate a first data form, wherein the first data form comprises a plurality of sample data, and the plurality of sample data comprises the first sample data;

[0008] Obtaining a test program created based on a Spark computing engine, and performing business logic calculation on each sample data in the first data form through the test program to obtain a first calculation result corresponding to each sample data in the first data form;

[0009] Synchronizing the first data form to a Hive data warehouse to obtain a second data form, and performing business logic calculation on each sample data in the second data form through a preset big data algorithm program to obtain a second calculation result corresponding to each sample data in the second data form;

[0010] Comparing the first calculation result and the second calculation result to obtain a comparison result corresponding to each sample data;

[0011] determine correctness of the big data algorithm program on the business logic calculation according to the checking result.

[0012] According to some embodiments of the present application, the method for testing program based on big data order generation further comprises:

[0013] obtaining second sample data of a second business product from a business database;

[0014] The step of storing the first sample data into a preset relational database to generate a first data form comprises:

[0015] storing the first sample data and the second sample data into a preset relational database to generate a first data form.

[0016] According to some embodiments of the present application, the step of obtaining second sample data of a second business product from a business database comprises:

[0017] obtaining sample data of preset fields of the second business product from the business database according to a preset rule to obtain the second sample data, wherein the preset fields are matched with fields in the product data configuration table.

[0018] According to some embodiments of the present application, the step of generating the first sample data of the first business product based on the product data configuration table comprises:

[0019] obtaining a preset Python data generation program;

[0020] generating the first sample data of the first business product based on the product data configuration table and using the Python data generation program.

[0021] According to some embodiments of the present application, the step of checking the first calculation result and the second calculation result to obtain a checking result corresponding to each sample data comprises:

[0022] checking the first calculation result and the second calculation result to obtain a difference value between the first calculation result and the second calculation result corresponding to each sample data;

[0023] generating a checking result corresponding to each sample data according to a preset result template and the difference value.

[0024] According to some embodiments of the present application, the business logic calculation includes a plurality of sub-logic calculations, and the first calculation result and the second calculation result each include a plurality of sub-calculations;

[0025] The checking processing of the first calculation result and the second calculation result obtains a difference value between the first calculation result and the second calculation result corresponding to each sample data.

[0026] The checking processing of the first calculation result and the second calculation result obtains a difference value between the first calculation result and the second calculation result corresponding to each sample data.

[0027] According to some embodiments of the present application, the business logic calculation includes a plurality of sub-logic calculations, and the first calculation result and the second calculation result each include a plurality of sub-calculations;

[0028] The checking processing of the first calculation result and the second calculation result obtains a difference value between the first calculation result and the second calculation result corresponding to each sample data.

[0029] To achieve the above object, a second aspect of the embodiments of the present application provides a program testing device based on big data single generation, which comprises:

[0030] A data acquisition module is configured to acquire a product data configuration table of a first business product and generate first sample data of the first business product based on the product data configuration table.

[0031] A form generation module is configured to store the first sample data to a preset relational database and generate a first data form, wherein the first data form includes a plurality of sample data, and the plurality of sample data includes the first sample data.

[0032] A first calculation module is configured to acquire a test program created based on a Spark calculation engine, perform business logic calculation on each sample data in the first data form by using the test program, and obtain a first calculation result corresponding to each sample data in the first data form.

[0033] A second calculation module is configured to synchronize the first data form to a Hive data warehouse to obtain a second data form, perform business logic calculation on each sample data in the second data form by using a preset big data algorithm program, and obtain a second calculation result corresponding to each sample data in the second data form.

[0034] A result checking module is configured to perform checking processing on the first calculation result and the second calculation result to obtain a checking result corresponding to each sample data.

[0035] a processing module configured to determine correctness of the big data algorithm program on the business logic calculation according to the comparison result.

[0036] To achieve the above object, a third aspect of embodiments of the present application provides an electronic device, which comprises a memory, a processor, a computer program stored in the memory and executable on the processor, and the computer program is executed by the processor to implement the method of the first aspect.

[0037] To achieve the above object, a fourth aspect of embodiments of the present application provides a storage medium, which is a computer readable storage medium, for computer readable storage, and the storage medium stores one or more computer programs, and the one or more computer programs are executable by one or more processors to implement the method of the first aspect.

[0038] The present application provides a program testing method, device, electronic device and computer readable storage medium based on big data single generation. The program testing method based on big data single generation acquires a product data configuration table of a first business product, generates first sample data of the first business product based on the product data configuration table, stores the first sample data to a preset relational database to generate a first data form, acquires a test program created based on a Spark computing engine, performs business logic calculation on each sample data in the first data form through the test program to obtain a first calculation result corresponding to each sample data, synchronizes the first data form to a Hive data warehouse to obtain a second data form, performs business logic calculation on each sample data in the second data form through a preset big data algorithm program to obtain a second calculation result corresponding to each sample data, compares the first calculation result and the second calculation result, and finally determines correctness of the big data algorithm program on the business logic calculation according to a comparison result corresponding to each sample data. The present application generates single based on the product data configuration table of the business product, constructs the first data form stored in the relational database and the second data form stored in the Hive data warehouse, then performs business logic calculation on the first data form in the relational database and the second data form in the Hive data warehouse through the test program created by the Spark computing engine and the preset big data algorithm program respectively, obtains the first calculation result and the second calculation result, finally determines correctness of the big data algorithm program on the business logic calculation according to the two calculation results, and realizes testing of the algorithm program. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a flowchart of a program testing method based on big data single generation provided by embodiments of the present application;

[0040] Figure 2 is a flow diagram of a program test method based on big data order creation according to another embodiment of the present application;

[0041] Figure 3 is a flow diagram of a program test method based on big data order creation according to another embodiment of the present application;

[0042] Figure 4 is a flow diagram of a program test method based on big data order creation according to another embodiment of the present application;

[0043] Figure 5 is a structural diagram of a program test device based on big data order creation according to an embodiment of the present application;

[0044] Figure 6 is a hardware structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0046] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0047] First, several terms involved in the present application are analyzed:

[0048] Spark computing engine, a fast general-purpose computing engine designed for large-scale data processing. Spark is a general-purpose parallel framework similar to Hadoop MapReduce open sourced by the AMP laboratory of the University of California, Berkeley. Spark has the advantages of Hadoop MapReduce; but unlike MapReduce, the intermediate output results in Job can be saved in memory, so there is no need to read and write HDFS. Therefore, Spark can better adapt to MapReduce algorithms such as data mining and machine learning that require iteration.

[0049] Hive data warehouse, a data warehouse built on the running framework of Hadoop big data software system, can map structured data files into tables and provide SQL query functions, and the SQL statement used for query is converted into a MapReduce job and then submitted to Hadoop for running.

[0050] With the increasing business volume of enterprises, the number of new business products of enterprises is gradually increasing every month. Before each new business product goes online, the business logic algorithm of the product needs to be tested. For example, in the life insurance field, different effective premium algorithms are used according to different business types and business properties of each insurance product. Therefore, the effective premium algorithm of the product needs to be tested before the new insurance product goes online, that is, the big data algorithm program of the effective premium is tested.

[0051] However, for new business products that have not gone online, there is no inventory data that can be used for corresponding program testing.

[0052] Based on this, the embodiment of the application provides a program testing method and device based on big data single creation, electronic equipment and computer readable storage medium, which can test the business logic big data algorithm program of the new business product.

[0053] The program testing method and device based on big data single creation, electronic equipment and computer readable storage medium provided by the embodiment of the application are specifically explained by the following embodiment. First, the program testing method based on big data single creation in the embodiment of the application is described.

[0054] The embodiment of the application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (Artificial Intelligence, AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. Theory, method, technology and application system.

[0055] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology and machine learning / deep learning, etc.

[0056] The program test method based on big data order generation provided by the embodiments of the present application can be applied to a terminal, can be applied to a server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server side can be configured as a separate physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform; and the software can be an application program for implementing the program test method based on big data order generation, but is not limited to the above forms.

[0057] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as a program module. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0058] Please refer to Figure 1 , Figure 1 A flowchart of a program test method based on big data order generation provided by the embodiments of the present application is shown. As shown in Figure 1 , the program test method based on big data order generation includes but is not limited to steps S110 to S160:

[0059] Step S110, obtaining a product data configuration table of a first service product, and generating first sample data of the first service product based on the product data configuration table.

[0060] Step S120, storing the first sample data to a preset relational database to generate a first data form, the first data form including multiple sample data, and the multiple sample data including the first sample data.

[0061] Step S130: Obtain a test program created based on the Spark computing engine, and perform business logic calculations on each sample data in the first data form through the test program to obtain the first calculation result corresponding to each sample data in the first data form.

[0062] Step S140: Synchronize the first data form to the Hive data warehouse to obtain the second data form, and perform business logic calculations on each sample data in the second data form using a preset big data algorithm program to obtain the second calculation result corresponding to each sample data in the second data form.

[0063] Step S150: Verify the first calculation result and the second calculation result to obtain the verification result corresponding to each sample data.

[0064] Step S160: Determine the correctness of the big data algorithm program in the business logic calculation based on the verification result.

[0065] For example, in the life insurance field, the effective sum assured refers to the maximum amount that an insurance company must pay when an insured event with the maximum payout under the insurance contract occurs. Each insurance product employs a different effective sum assured algorithm due to its different business type and nature. Therefore, before a new insurance product is launched, the effective sum assured algorithm program for the new product needs to be tested. In the big data-based policy creation testing method provided in this application embodiment, the first business product is the new business product, and the second business product is the old business product.

[0066] Understandably, the product data configuration table includes the business fields required for business logic calculations. For example, if the first business product is an insurance product, and the business logic calculation is the calculation of the effective sum insured, then the product data configuration table includes the business fields required for the calculation of the effective sum insured: each type of insurance in the first business product, each policy status, each liability, and each segment, etc.

[0067] In some embodiments, see Figure 3 , Figure 3 This illustration shows a flowchart of a program testing method for creating orders based on big data, as provided in an embodiment of this application. Figure 3 As shown, the generation of the first sample data of the first business product based on the product data configuration table includes, but is not limited to, steps S310 to S320:

[0068] Step S310: Obtain the preset Python data generation program.

[0069] In step S320, the first sample data of the first service product is generated based on the product data configuration table and by using the Python data generation program.

[0070] It can be understood that the data generation program written based on the Python programming language is acquired, and the first sample data of the first service product is automatically generated according to the product data configuration table of the first service product by using the Python data generation program.

[0071] In one specific embodiment, the Python data generation program generates the first sample data conforming to the constraint condition of each field in the product data configuration table according to the constraint condition.

[0072] It can be understood that the constraint condition is used to limit the value type and range of each field in the product data configuration table, so that the Python data generation program generates the first sample data conforming to the actual application rule, and the correctness of the big data algorithm program in the business logic calculation is tested based on the first sample data conforming to the actual application rule, which can ensure the rationality and accuracy of the test.

[0073] It can be understood that in step S120, the preset relational database can be an Oracle, a SqlServer or a Mysql relational database.

[0074] In some embodiments, the plurality of sample data further includes second sample data of a second service product, please refer to Figure 2 , Figure 2 A flowchart of a program testing method based on big data single generation provided by an embodiment of the present application is shown, as shown in Figure 2 Before step S120, the method further includes but is not limited to step S210.

[0075] In step S210, second sample data of a second service product is acquired from a business database.

[0076] The step S120 includes but is not limited to step S220.

[0077] In step S220, the first sample data and the second sample data are stored into a preset relational database to generate a first data form.

[0078] It can be understood that the business logic calculation of the first business product can cover other business products, that is, the business logic calculation on the new business product can involve the old business product, and if the correlation analysis in the algorithm program development stage is not in place, it will cause the old business product calculation error, therefore, by obtaining the business data of the old business product from the existing production environment, that is, obtaining the second sample data of the second business product from the business database, the first sample data and the second sample data are stored in the preset relational database together, so that the sample data in the generated first data form covers the new and old business products, thereby comprehensively guaranteeing the code quality of the algorithm program.

[0079] It should be further pointed out that the program test method based on big data single generation can first obtain the second sample data of the old business product from the business database, then store the second sample data in the preset relational database to generate the first data form, and then add the first sample data of the new business product to the first data form based on the product data configuration table of the new business product, so the first data form includes multiple sample data of new and old business products.

[0080] In one specific embodiment, the step S210 includes:

[0081] The second sample data of the second business product is obtained from the multiple business databases through a join query.

[0082] It can be understood that the data of the business product can be stored separately through multiple business databases, therefore, the second sample data corresponding to the second business product needs to be obtained from multiple business databases through a join query.

[0083] In one specific embodiment, the step S210 includes:

[0084] The second sample data of multiple second business products is obtained from the business database.

[0085] It can be understood that according to the business scope covered by the business logic calculation on the first business product, the second sample data of multiple second business products is obtained from the business database. For example, the effective premium calculation of the first business product can affect all insurance products of the current life insurance, and then the second sample data of all insurance products is obtained from the corresponding business database.

[0086] In some embodiments, the step S210 further includes:

[0087] According to a preset rule, sample data of a preset field of the second business product is obtained from the business database to obtain the second sample data, wherein the preset field matches the field in the product data configuration table.

[0088] It can be understood that each business product corresponds to a large amount of business data. If all the business data of the business product is taken as the second sample data for program testing, the testing efficiency will be greatly reduced. Therefore, the part of the business data of the first business product that meets the preset rule is taken as the second sample data from the business database, that is, the part of the business data of the first business product is taken as the second sample data from the business database, which can reduce the testing time and improve the testing efficiency.

[0089] It can be understood that the business logic calculation of the business product often does not require all the field data of the business product. Since the product data configuration table includes the business fields required for the business logic calculation, the business data matched with the fields in the product data configuration table is taken as the second sample data from the business database.

[0090] For example, the first business product is an insurance product, and the business logic calculation is effective premium calculation. The effective premium calculation may affect all current insurance products. The product data configuration table includes the business fields required for the effective premium calculation, such as each type of insurance in the first business product, each policy state, each liability, and each segment. Specifically, the part of the business data of all insurance products that meets the preset rule and is matched with the fields in the product data configuration table is taken as the second sample data from the business database. The first sample data and the second sample data are stored in the preset relational database to obtain the first data form. The part of the business data of the second business product that meets the preset rule is taken as the second sample data from the business database in a sampling manner. The sample data in the first data form covers the new and old insurance products, and the number of sample data and useless data in the first data form is reduced, which can guarantee the code quality of the big data algorithm program, reduce the testing time, and improve the testing efficiency.

[0091] It can be understood that in step S130, the test program created based on the Spark computing engine and the preset big data algorithm program are used for the same business logic calculation of the first business product.

[0092] It can be understood that in step S140, the preset big data algorithm program refers to an algorithm program written by a developer for performing corresponding business logic calculation on data in the Hive data warehouse, and is also a test object of the program testing method for single generation based on big data provided by the present application. The traditional database multi-threaded batch processing is not capable of dealing with millions of business data, and by deploying the big data algorithm program to perform business product corresponding business logic calculation in the Hive data warehouse, the ability of the big data cluster to analyze and process massive data can be fully utilized to provide business logic calculation speed. Therefore, testing the algorithm program of the first business product in business logic is to test the preset big data algorithm program, and the big data algorithm program is applied to the Hive data warehouse, so that the first data table corresponding to the first business product needs to be synchronized to the Hive data warehouse to obtain a second data table, so as to perform business logic calculation on the second data table through the big data algorithm program.

[0093] For example, in actual application, based on the business logic calculation requirement of the new business product provided by the business personnel, the developer codes a big data algorithm program for performing business logic calculation in the Hive data warehouse. Before the new business product goes online, the tester constructs a test program for performing business logic calculation in the relational database based on the same business logic calculation requirement through the Spark computing engine. Thus, based on the test program and a data table, the program testing of the big data algorithm program can be completed.

[0094] In some embodiments, please refer to Figure 4 , Figure 4 The sub-step flowchart of step S150 in Figure 1 is shown, as shown in Figure 4 , the step S150 includes but is not limited to steps S410 to S420:

[0095] Step S410, the first calculation result and the second calculation result are collated to obtain the difference value between the first calculation result and the second calculation result corresponding to each sample data.

[0096] Step S420, according to the preset result template and the difference value, a collation result corresponding to each sample data is generated.

[0097] It should be noted that the test program and big data algorithm program perform business logic calculations on the first data form in the relational database and the second data form in the Hive data warehouse, respectively, to obtain the first and second calculation results for each sample data in the data forms. Then, the difference between the first and second calculation results is calculated to obtain the difference value between the first and second calculation results for each sample data. Based on a preset result template, the corresponding field data in each sample data is obtained. Based on the field data and the difference value, a verification result for each sample data is generated. It is understandable that generating the verification result for each sample data according to the preset result template further visualizes the program test results.

[0098] In some embodiments, the business logic calculation includes multiple sub-logic calculations, and both the first calculation result and the second calculation result include multiple sub-calculation results;

[0099] The step of verifying the first calculation result and the second calculation result to obtain the verification result corresponding to each sample data includes:

[0100] The multiple sub-calculation results of the first calculation result and the second calculation result are checked and processed respectively to obtain the difference value between the multiple sub-calculation results corresponding to each sample data.

[0101] Understandably, big data algorithm programs involve multiple logical steps in their business logic calculations. These steps are then divided into sub-logical calculations, each corresponding to a sub-result. In other words, the first and second calculation results for each sample data set each have multiple sub-results. By determining the differences between these sub-results for each sample data set, we can intuitively identify the discrepancies in each sub-calculation—that is, the differences between the test program and the big data algorithm program across multiple logical steps. This allows us to quickly pinpoint the logical step in which the big data algorithm program encountered an error, facilitating subsequent code updates.

[0102] For example, suppose the big data algorithm program corresponding to the business logic calculation is:

[0103] Fields of sample data Perform a weighted summation to obtain the value. For fields and Perform multiplication to obtain the numerical value. logarithmic values and Perform summation to obtain the calculation result. .

[0104] According to the logical steps of the big data algorithm procedure on the business logic calculation, the business logic calculation is divided into three sub-logic calculations:

[0105] (1) weighted sum is performed on the fields of the sample data to obtain a numerical value .

[0106] (2) product processing is performed on the fields and to obtain a numerical value .

[0107] (3) sum processing is performed on the numerical values and to obtain a numerical value .

[0108] Therefore, the calculation result corresponds to three sub-calculation results, the sub-calculation result in the first sub-logic calculation is the numerical value , the sub-calculation result in the second sub-logic calculation is the numerical value , and the sub-calculation result in the third sub-logic calculation is the numerical value .

[0109] In some embodiments, the synchronizing the first data form to the Hive data warehouse to obtain a second data form comprises:

[0110] synchronizing the first data form to the Hive data warehouse by using a data processing tool Sqoop to obtain a second data form.

[0111] It should be understood that Sqoop is an open source tool that can be used for data transfer between the Hive data warehouse and a traditional relational database. It can import data in a relational database into the distributed file system of the Hive data warehouse, or import data in the distributed file system of the Hive data warehouse into a relational database.

[0112] It can be understood that the data processing tool Sqoop can quickly synchronize a large amount of sample data in the first data form to the Hive data warehouse, thereby improving the overall efficiency of program testing.

[0113] In some embodiments, after the verification result corresponding to each piece of sample data is generated according to the preset result template and the difference value, the method further comprises:

[0114] storing the verification result corresponding to each piece of sample data to the relational database.

[0115] It can be understood that storing the collation result into the relational database can be used as a test result of a big data algorithm program of business logic calculation, for data comparison in a subsequent program updating process.

[0116] The program testing method based on big data policy generation provided by the embodiments of the present application is described below through a specific embodiment:

[0117] The method is applied to the insurance field and is used for testing a big data algorithm program of an effective insurance amount, and the method specifically comprises:

[0118] Sample data of an old insurance product is obtained from a business database, and the sample data is stored into a preset relational database to generate a first data form, sample data of a new insurance product is added into the first data form according to a product data configuration table of the new insurance product through a Python data generation program, and the first data form covered with the sample data of the new and old insurance products is synchronized into a Hive data warehouse.

[0119] A test program based on a Spark computing engine and a preset big data algorithm program perform business logic calculation on the first data form in the relational database and the second data form in the Hive data warehouse respectively, to obtain a first calculation result and a second calculation result corresponding to each sample data in the data form, determine a difference value between the first calculation result and the second calculation result corresponding to each sample data, and if the difference value is not 0, it is determined that there is an error or a vulnerability in the business logic calculation of the big data algorithm program.

[0120] The application provides a program testing method based on big data single generation. The program testing method based on big data single generation obtains a product data configuration table of a first business product, generates first sample data of the first business product based on the product data configuration table, stores the first sample data into a preset relational database to generate a first data form, obtains a test program created based on a Spark computing engine, performs business logic calculation on each piece of sample data in the first data form through the test program to obtain a first calculation result corresponding to each piece of sample data, synchronizes the first data form to a Hive data warehouse to obtain a second data form, performs business logic calculation on each piece of sample data in the second data form through a preset big data algorithm program in the Hive data warehouse to obtain a second calculation result corresponding to each piece of sample data, checks the first calculation result and the second calculation result, and finally determines the correctness of the big data algorithm program in business logic calculation according to a checking result corresponding to each piece of sample data. The embodiment of the application generates a single based on the product data configuration table of the business product, constructs the first data form stored in the relational database and the second data form stored in the Hive data warehouse, then performs business logic calculation on the first data form in the relational database and the second data form in the Hive data warehouse through the test program created by the Spark computing engine and the preset big data algorithm program respectively, obtains the first calculation result and the second calculation result, finally determines the correctness of the big data algorithm program in business logic calculation according to the two calculation results, and realizes the testing of the big data algorithm program.

[0121] See Figure 5 The embodiment of the application further provides a program testing device 100 based on big data single generation, which comprises:

[0122] A data obtaining module 110 is used for obtaining a product data configuration table of a first business product, and generating first sample data of the first business product based on the product data configuration table.

[0123] A form generating module 120 is used for storing the first sample data into a preset relational database to generate a first data form, wherein the first data form comprises a plurality of pieces of sample data, and the plurality of pieces of sample data comprise the first sample data.

[0124] A first calculation module 130 is used for obtaining a test program created based on a Spark computing engine, and performing business logic calculation on each piece of sample data in the first data form through the test program to obtain a first calculation result corresponding to each piece of sample data in the first data form.

[0125] The second computing module 140 is configured to synchronize the first data form to a Hive data warehouse to obtain a second data form, and perform business logic calculation on each sample data in the second data form through a preset big data algorithm program to obtain a second calculation result corresponding to each sample data in the second data form.

[0126] The result checking module 150 is configured to check the first calculation result and the second calculation result to obtain a checking result corresponding to each sample data.

[0127] The processing module 160 is configured to determine the correctness of the big data algorithm program in the business logic calculation according to the checking result.

[0128] The application provides a program testing device based on big data form generation. The program testing device based on big data form generation obtains a product data configuration table of a first business product through a data acquisition module, and generates first sample data of the first business product based on the product data configuration table. Then, a form generation module stores the first sample data to a preset relational database to generate a first data form. A first computing module obtains a test program created based on a Spark computing engine, and performs business logic calculation on each sample data in the first data form through the test program to obtain a first calculation result corresponding to each sample data. A second computing module synchronizes the first data form to a Hive data warehouse to obtain a second data form, and performs business logic calculation on each sample data in the second data form through a preset big data algorithm program in the Hive data warehouse to obtain a second calculation result corresponding to each sample data. A result checking module checks the first calculation result and the second calculation result. Finally, a processing module determines the correctness of the big data algorithm program in the business logic calculation according to a checking result corresponding to each sample data. The application generates forms based on the product data configuration table of a business product, constructs a first data form stored in a relational database and a second data form stored in a Hive data warehouse, and then performs business logic calculation on the first data form in the relational database and the second data form in the Hive data warehouse through a test program created by a Spark computing engine and a preset big data algorithm program, respectively, to obtain a first calculation result and a second calculation result. Finally, the correctness of the big data algorithm program in the business logic calculation is determined according to the two calculation results, and the big data algorithm program is tested.

[0129] It should be noted that the information interaction, execution process and the like between the modules of the above device are based on the same concept as the method embodiments of the application, and the specific functions and the technical effects brought by the same can be referred to the method embodiments part, which will not be described here.

[0130] Please refer toFigure 6 , Figure 6 The hardware structure of an electronic device provided by an embodiment of the present application is shown, and the electronic device comprises:

[0131] The processor 210 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute a related computer program to implement the technical solutions provided by the embodiments of the present application.

[0132] The memory 220 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 220 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 220 and are called and executed by the processor 210 to implement the program test method based on big data order generation of the embodiments of the present application.

[0133] The input / output interface 230 is used to realize information input and output.

[0134] The communication interface 240 is used to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.). The bus 250 is used to transmit information between each component (for example, the processor 210, the memory 220, the input / output interface 230, and the communication interface 240) of the device.

[0135] The processor 210, the memory 220, the input / output interface 230, and the communication interface 240 are connected to each other by the bus 250 to realize the communication connection between the device.

[0136] The embodiments of the present application further provide a storage medium, which is a computer readable storage medium, and is used for computer readable storage. The storage medium stores one or more computer programs, and the one or more computer programs can be executed by one or more processors to implement the above-mentioned program test method based on big data order generation.

[0137] The memory, as a computer-readable storage medium, can be used to store software programs and computer-executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely from the processor, which can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0138] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0139] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0140] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware or appropriate combinations thereof.

[0141] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0142] The terms "first", "second", "third", "fourth" and the like used in the description of the present application and the above drawings (if any) are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0143] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B, and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, and c can be single or multiple.

[0144] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or can be combined into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between devices or units, which can be electrical, mechanical or other forms.

[0145] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0146] In addition, each functional unit in each embodiment of the application can be integrated into one processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0147] The unit formed by the collection can be stored in a computer readable storage medium if it is realized in the form of a software function unit and sold or used as an independent product. Based on such an understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0148] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A program test method based on big data single creation, characterized by, The method comprises: obtaining product data configuration table of first service product, and generating first sample data of the first service product based on the product data configuration table; obtaining second sample data of second service product from service database; storing the first sample data and the second sample data to preset relational database to generate first data form, the first data form comprising multiple sample data, the multiple sample data comprising the first sample data and the second sample data; obtaining test program created based on Spark computing engine, and performing business logic calculation on each sample data in the first data form through the test program to obtain first calculation result corresponding to each sample data in the first data form, the business logic calculation comprising multiple sub-logic calculations, and the first calculation result comprising multiple sub-calculations; synchronizing the first data form to Hive data warehouse to obtain second data form, and performing business logic calculation on each sample data in the second data form through preset big data algorithm program to obtain second calculation result corresponding to each sample data in the second data form, the second calculation result comprising multiple sub-calculations; performing checking processing on the first calculation result and the multiple sub-calculations of the second calculation result respectively to obtain difference value between the multiple sub-calculations corresponding to each sample data; generating checking result corresponding to each sample data according to preset result template and the difference value; determining correctness of the big data algorithm program in the business logic calculation according to the checking result.

2. The big data based program test method of claim 1, wherein, The method comprises: obtaining sample data of preset field of the second service product from the service database according to preset rule to obtain the second sample data, wherein the preset field matches the field in the product data configuration table. 3.The big data based program test method of claim 1, wherein, The method comprises: obtaining preset Python data generation program; generating the first sample data of the first service product based on the product data configuration table and using the Python data generation program.

4. The big data based program test method of claim 1, wherein, The method comprises: synchronizing the first data form to Hive data warehouse through data processing tool Sqoop to obtain the second data form.

5. A program testing device based on big data single creation, characterized by, The device is used for implementing the method of any one of claims 1-4, and the device comprises: a data acquisition module configured to obtain product data configuration table of first service product, and generate first sample data of the first service product based on the product data configuration table; a form generation module configured to store the first sample data to preset relational database to generate first data form, the first data form comprising multiple sample data, the multiple sample data comprising the first sample data; The first computing module is configured to obtain a test program created based on a Spark computing engine, and perform business logic computation on each sample data in the first data form through the test program to obtain a first computing result corresponding to each sample data in the first data form; The second computing module is configured to synchronize the first data form to a Hive data warehouse to obtain a second data form, and perform business logic computation on each sample data in the second data form through a preset big data algorithm program to obtain a second computing result corresponding to each sample data in the second data form; The result checking module is configured to check the first computing result and the second computing result to obtain a checking result corresponding to each sample data. The processing module is configured to determine the correctness of the big data algorithm program in the business logic computation according to the checking result.

6. An electronic device, comprising: comprise: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program, and the computer program is executed by the at least one processor to enable the at least one processor to execute the program testing method based on big data order creation according to any one of claims 1 to 4.

7. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the program testing method based on big data order creation according to any one of claims 1 to 4.

Images