Big Data Application Development Method, Device, Computer Equipment and Storage Medium

Abstract

The present application relates to a big data application development method, apparatus, computer device, and storage medium. The method includes: obtaining an application development request, creating an application development project through multi-tenant technology, and allocating a corresponding workspace for the application development project; publishing an application development task corresponding to the development project, and obtaining task code corresponding to the application development task; when a test data request is received, searching for historical data, and transferring application test data corresponding to the historical data to the workspace; testing a big data application formed by the task code through the application test data, and obtaining a big data application that passes the test. In the present application, an application development project and a workspace for application development are constructed through multi-tenant technology according to the application development request, and the isolation and sharing of data in the application development project are ensured through the workspace, thereby reducing the server investment in the development process. The efficiency of the development of the big data application is effectively ensured through the cooperation of the development and testing processes.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to a method, device, computer equipment and storage medium for developing big data applications. Background Art

[0002] With the development of computer technology, big data technology has emerged. It refers to a collection of data that cannot be captured, managed, and processed by conventional software tools within a certain time range, and is a massive, high-growth, and diverse information asset that requires new processing models to have stronger decision-making power, insight discovery ability, and process optimization ability. And for some big data, its confidentiality and security need to be ensured. For example, for government affairs big data, it is transmitted using a private network, and extremely high requirements are imposed on data confidentiality and security. Therefore, when developing applications or governing data for such data, isolating the development and test environment from the production environment has become a necessity in the construction of the development platform.

[0003] At the same time, due to high security requirements, the development of such big data applications is usually deployed in the private cloud mode at present. In the private cloud, desensitization operations such as online masking, transformation, character replacement, and random replacement can be performed on sensitive data in the database in the way of processing database files, so as to achieve the confidentiality effect of enterprise core data.

[0004] However, compared with the public cloud, the resources for application development in the private cloud mode are limited, and the elastic expansion ability is weak. The test environment and the production environment are separately deployed, including the supporting backend database, big data platform, etc., all of which require multiple sets of environments. Therefore, from the perspective of the resource occupancy of the platform management node or the reservation of computing / storage resources, the above solutions require a large amount of server resources, thus affecting the utilization efficiency of server resources. Summary of the Invention

[0005] Based on this, in view of the above technical problems, it is necessary to provide a method, device, computer equipment and storage medium for developing big data applications that can improve the utilization efficiency of server resources.

[0006] A method for developing big data applications, the method includes:

[0007] Obtain an application development request, where the application development request is used to request the development of a big data application;

[0008] According to the application development request, create an application development project through multi-tenant technology, and allocate a corresponding working space for the application development project;

[0009] Publish the application development tasks corresponding to the development project, and obtain the task codes corresponding to the application development tasks;

[0010] When a test data request is received, search for the historical data corresponding to the test data request, and transfer the application test data corresponding to the historical data to the workspace;

[0011] Test the big data application composed of the task code with the application test data to obtain the big data application that passes the test.

[0012] A big data application development device, the device includes:

[0013] A request acquisition module, configured to acquire an application development request, where the application development request is used to request the development of a big data application;

[0014] A project construction module, configured to create an application development project through multi-tenant technology according to the application development request, and allocate a corresponding workspace for the application development project;

[0015] A task publishing module, configured to publish the application development task corresponding to the development project, and acquire the task code corresponding to the application development task;

[0016] A data transfer and storage module, configured to, when a test data request is received, search for the historical data corresponding to the test data request, and transfer the application test data corresponding to the historical data to the workspace;

[0017] An application acquisition module, configured to test the big data application composed of the task code with the application test data to obtain the big data application that passes the test.

[0018] A computer device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0019] Acquire an application development request, where the application development request is used to request the development of a big data application;

[0020] According to the application development request, create an application development project through multi-tenant technology, and allocate a corresponding workspace for the application development project;

[0021] Publish the application development task corresponding to the development project, and acquire the task code corresponding to the application development task;

[0022] When a test data request is received, search for the historical data corresponding to the test data request, and transfer the application test data corresponding to the historical data to the workspace;

[0023] Test the big data application composed of the task code with the application test data to obtain the big data application that passes the test.

[0024] A computer-readable storage medium stores a computer program thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0025] Obtain an application development request for requesting the development of a big data application;

[0026] According to the application development request, create an application development project through multi-tenant technology and allocate a corresponding workspace for the application development project;

[0027] Publish the application development tasks corresponding to the development project and obtain the task codes corresponding to the application development tasks;

[0028] When a test data request is received, search for the historical data corresponding to the test data request and transfer the application test data corresponding to the historical data to the workspace;

[0029] Test the big data application formed by the task codes through the application test data to obtain a big data application that passes the test.

[0030] The above big data application development method, device, computer device, and storage medium obtain an application development request for requesting the development of a big data application; according to the application development request, create an application development project through multi-tenant technology and allocate a corresponding workspace for the application development project; publish the application development tasks corresponding to the development project and obtain the task codes corresponding to the application development tasks; when a test data request is received, search for the historical data corresponding to the test data request and transfer the application test data corresponding to the historical data to the workspace; test the big data application formed by the task codes through the application test data to obtain a big data application that passes the test. In this application, an application development project and an application development workspace are constructed through multi-tenant technology according to the application development request, and the isolation and sharing of data in the application development project are ensured through the workspace, thereby reducing the server investment during the development process. At the same time, during the development process, task codes required for big data application development are obtained by publishing application development tasks within the development project, application test data required for application testing is transferred by processing test data requests, and the efficiency of big data application development is effectively ensured through the cooperation of the development and testing processes. Brief Description of the Drawings

[0031] Figure 1 It is an application environment diagram of the big data application development method in an embodiment;

[0032] Figure 2 It is a flowchart of the big data application development method in an embodiment;

[0033] Figure 3Schematic flowchart of steps for creating a development project in an embodiment;

[0034] Figure 4 Schematic flowchart of steps for generating and storing application test data in an embodiment;

[0035] Figure 5 Schematic flowchart of steps for migrating application test data through a data migration task in an embodiment;

[0036] Figure 6 Schematic diagram of the structure of a workspace in an embodiment;

[0037] Figure 7 Business flowchart of a big data application development method in an embodiment;

[0038] Figure 8 Structural block diagram of a big data application development device in an embodiment;

[0039] Figure 9 Internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0040] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0041] The big data application development method provided by the present application can be applied to an application environment as shown in Figure 1 . Among them, the terminal 102 where the developer is located communicates with the development management server 104 through a network. The terminal 102 submits an application development request to the development management server, and the development management server 104 obtains the application development request. The application development request is used to request the development of a big data application. According to the application development request, an application development project is created through multi-tenant technology, and a corresponding workspace is allocated to the application development project. The application development task corresponding to the development project is published, and the corresponding task code fed back by the terminal 102 according to the application development task is obtained. When a test data request is received, the historical data corresponding to the test data request is searched, and the application test data corresponding to the historical data is stored in the workspace. The big data application composed of the task code is tested with the application test data, and the big data application that passes the test is obtained. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices, and the server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0042] In one embodiment, as shown in Figure 2As shown in the figure, a method for developing big data applications is provided. Taking the development management server 104 in Figure 1 as an example for illustration, the method includes the following steps:

[0043] Step 201: Obtain an application development request, which is used to request the development of a big data application.

[0044] The application development request is used to request the development of a big data application. Specifically, it is submitted by the terminal 102 to the development management server 104 to request the development management server 104, as a development platform, to provide technical support in aspects such as the production environment and the test environment for the development of the big data application. The application development request specifically includes data such as the required computing resources, storage resources, and the permission allocation of development roles. Big data refers to a data set that cannot be captured, managed, and processed by conventional software tools within a certain time range. It is a massive, high-growth-rate, and diverse information asset that requires a new processing mode to have stronger decision-making power, insight discovery ability, and process optimization ability. An application developed based on big data is a big data application.

[0045] Specifically, when a developer needs to develop a big data application, since it is necessary to coordinate the development environment, test environment, and production environment during the big data development process, an application development request can be sent to the development management server 104 to request the development management server 104 to provide corresponding space and resources for subsequent big data application development.

[0046] Step 203: According to the application development request, create an application development project through multi-tenancy technology and allocate a corresponding workspace for the application development project.

[0047] Among them, multi-tenancy technology, also known as multiple leasing technology, is a software architecture technology that realizes how to share the same system or program components in a multi-user environment and can ensure the isolation of data between users. A development project is a business entity based on the development management server 104 to divide and isolate resources. By adding computing resources, storage resources, and assigning different roles to members to the development project, users have different permissions in different services bound to the development project, and then use the resources. A project contains a unique resource queue, including the CPU (central processing unit), memory, storage space, and the priority is allocated between different resource queues. The workspace specifically refers to the overall space of the development, test, and production environment spaces created based on the development project and with resources allocated proportionally. The content of the space includes the CPU, memory, storage space, etc.

[0048] Specifically, since the development management server 104 can support multiple different big data application development projects simultaneously, in order to ensure the independence of each big data application and isolate the development data of different projects, corresponding development projects can be created through multi-tenant technology, and the isolation of data between the development projects of different users can be ensured by partitioning the workspaces. In one embodiment, it is also possible to set within the workspace that all code is only supported for editing in the development environment and cannot modify the code in the production environment, ensuring the data isolation between the development process and the production process.

[0049] Step 205: Publish the application development task corresponding to the development project and obtain the task code corresponding to the application development task.

[0050] The application development task is used to instruct the developers on the side of the terminal 102 to write code programs that implement the corresponding functions. The task code refers to the code written based on the application development task, which meets the requirements of the application development task and can implement the corresponding functions. During the development process, project members with development roles carry out application development according to specific business requirements, using computing frameworks supported by the big data platform, such as spark, flink, etc.

[0051] Specifically, the information of the application development task can be provided by the development management personnel, and then published through the development management server 104. Then, the developers on the side of the terminal 102 write the corresponding task code according to the requirements of the task and submit it to the development management server 104. The development management server 104 performs corresponding application development management on the completed task code. Specifically, the development management server can also perform corresponding functional reviews on the task code to ensure the usability of the code. When the review fails, feedback the corresponding review information to the developers so that they can rewrite the corresponding task code or make corresponding adjustments to this part of the task code. For the task code that passes the review, the development management server 104 can provide an overall description for the completed task code according to the description of the application development task, and at the same time set the application development task to the completed state, and transfer the completed task code to the corresponding data preparation channel for use in constructing big data applications in subsequent processes.

[0052] Step 207: When receiving a test data request, search for the historical data corresponding to the test data request and transfer the application test data corresponding to the historical data to the workspace.

[0053] The data test request is used to request the development management server 104 for test data used for the final test of the big data application. The historical data refers to the real historical data involved in the big data application. The application test data is the data used for application testing based on the historical data. The historical real data is saved in the corresponding database accessible to the development management server 104. In order to ensure the confidentiality of the historical data, the developer cannot directly access this part of the data in the workspace.

[0054] Specifically, the development management server 104 is also used to provide test data support for the application development process. Since big data applications are created based on big data, a large amount of data is usually required for testing during the application testing process. At the same time, in some big data applications, the confidentiality of the test data used also needs to be guaranteed. At this time, the development management server 104 can first obtain this part of the historical data, and then perform corresponding data desensitization processing on the historical data to deform the data of these sensitive information through desensitization rules, so as to achieve reliable protection of sensitive privacy data. The desensitized test data is then saved to the workspace for use in the subsequent application testing process of the big data application.

[0055] Step 209: Test the big data application composed of the task code by applying the test data to obtain the big data application that passes the test.

[0056] The big data application is constructed based on the task code submitted by the developer to the development management server 104 .

[0057] Specifically, after the development of the big data application is completed, relevant test verification and tuning are required to ensure the correctness and efficiency of the application as much as possible. When the development management server 104 obtains the task code corresponding to the application development task, it will save these codes to the corresponding storage location. When the developer at the development end determines that all application development tasks are completed, the complete big data application can be built according to these codes in the development environment. The built big data application will be submitted to the test environment of the development management server 104, and then the final application verification and testing will be completed based on the application test data provided by the development management server 104, and a big data application that passes the test will be obtained. Specifically, in one of the embodiments, in the development process of the big data application, the application test data is first obtained, and then the application development is carried out, and then the test is carried out. The big data modeling development is different from the application development. This application is mainly to realize the data middle platform or data warehouse, a platform for data modeling development, so the development process is actually the process of data governance, standardization, analysis, and processing, so it is all based on the original data for processing. Therefore, first apply for a part of real data as the analyzed test data, and then develop the modeling and build the big data application.

[0058] The above big data application development method includes: obtaining an application development request for requesting the development of a big data application; according to the application development request, creating an application development project through multi-tenant technology and allocating a corresponding workspace for the application development project; publishing the application development tasks corresponding to the development project and obtaining the task codes corresponding to the application development tasks; when receiving a test data request, searching for the historical data corresponding to the test data request and transferring the application test data corresponding to the historical data to the workspace; testing the big data application composed of the task codes with the application test data to obtain the big data application that passes the test. In this application, an application development project and the workspace for application development are constructed through multi-tenant technology according to the application development request, and the isolation and sharing of data in the application development project are ensured through the workspace, thereby reducing the server investment in the development process. At the same time, during the development process, the task codes required for big data application development are obtained by publishing application development tasks within the development project, the application test data required for application testing is transferred by processing the test data request, and the efficiency of big data application development is effectively ensured through the coordination of the development and testing processes.

[0059] In one embodiment, as Figure 3 shown, step 203 includes:

[0060] Step 302, obtaining the resource information and role assignment information corresponding to the big data application according to the application development request.

[0061] Step 304, creating an application development project through multi-tenant technology, allocating a corresponding workspace for the application development project according to the resource information, and allocating corresponding project roles for the developers of the application development project according to the role assignment information.

[0062] Among them, the resource information specifically refers to the computing resources and storage resources required during the development of the big data application. The role assignment information specifically refers to the roles of each developer involved in the development process in the development project. By allocating roles, the access permissions of different developers in the project can be determined. Specifically, the personnel that the role assignment information can allocate include project managers, project developers, project operation and maintenance personnel, etc. The project manager can query the overall progress of the development during the project development process, publish the code writing tasks for application development, and publish the finally written and tested big data application, etc. The project developers are mainly responsible for writing the codes related to the big data application based on the tasks issued in the project during the project development process, and the project operation and maintenance personnel are mainly responsible for the operation and maintenance during the big data application development process and can also publish the big data application that passes the test.

[0063] Specifically, during the process of establishing a task process, the development management server 104 can obtain the resource information and role assignment information corresponding to the big data application according to the application development request; then create an application development project through multi-tenant technology, allocate a corresponding workspace for the application development project according to the resource information, and allocate corresponding project roles for the developers of the application development project according to the role assignment information. By adding computing resources, storage resources, and members with different roles to the development project, developers have different permissions in different services bound to the project. Furthermore, the resources are used. Among them, the process of allocating resources for the development project is to allocate a computing space similar to cloud resources for this project; and dividing the workspace based on the allocated resource information means that within this specified resource, according to business needs, the space sizes of each environment are allocated. In one embodiment, it can also be further divided within the workspace, for example, divided into a development environment, a production environment, and a test environment, etc., and each environment has its own resource configuration information. In this embodiment, by determining the resource information and role assignment information through the application development request, and then based on the role assignment information to perform the initial workspace allocation and the project role assignment corresponding to each task, the collaborative efficiency during the big data application development process can be effectively guaranteed, thereby improving the application development efficiency.

[0064] In one embodiment, before step 209, it further includes: auditing the task code according to the task requirements corresponding to the application development task, and generating and feedbacking the audit information corresponding to the task code.

[0065] Specifically, the development management server 104 can obtain the application code fed back by the application developer according to the application development task. At the same time, it can also review the submitted task code based on the workspace and generate corresponding review information. The review information is specifically fed back to the application developer side that submitted the task code, used to inform the application developer whether the provided task code is available. When it is not available, it also needs to prompt the application developer which part has a writing error, and prompt the application developer to rewrite it or adjust the corresponding task code. In this application, the workspace allocated for the application development project has the following advantages. First, all task codes only support editing in the development environment of the workspace, and cannot modify the code in the production environment. This ensures the isolation between the development environment and the production environment. At the same time, after the developer submits the task code, the development management server 104 will automatically audit the task code in the workspace to prevent high-risk operations such as drop table in the task code. In this embodiment, by auditing the task code and generating and feedbacking the audit information corresponding to the task code, the usability of the completed task code can be effectively guaranteed, thereby comprehensively improving the development efficiency of the big data application.

[0066] In one embodiment, before step 209, it further includes: when the audit information indicates that the audit is passed, the application development task and the task code are transferred into the environment scheduling system corresponding to the development project, and the environment scheduling system is used to provide a calling channel for the task code.

[0067] Among them, the workspace under the development project is further divided, for example, into a development environment, a production environment, a test environment, etc. Each environment has its own resource configuration information, which is used to support the development of different stages of the application. The environment scheduling system specifically includes a development environment scheduling system and a test environment scheduling system, corresponding to the development environment and the test environment respectively. The task code that has been written and passed the test can be transferred into the corresponding scheduling environment system according to the development stage of the application development task corresponding to the task code. Then, in the final application construction or application testing, the corresponding functions can be realized based on these task codes that have passed the test.

[0068] Specifically, after auditing the written task code, the task will automatically enter the development environment scheduling system or the test environment scheduling system after passing the audit. Then, in the subsequent process, the corresponding task code can be processed within the corresponding environment scheduling system. For example, the final big data application to be tested can be constructed according to the code in the development environment. In another embodiment, it is impossible to automatically import the written task code into the environment scheduling system. At this time, the written task code needs to be transferred and stored in the production environment in the production space by publishing the corresponding calling task in the workspace. In this embodiment, by transferring the application development task and the task code into the environment scheduling system corresponding to the development project, the subsequent application development and application testing work can be carried out more conveniently, thereby effectively improving the development efficiency of big data applications.

[0069] In one embodiment, as Figure 4 shown, step 207 includes:

[0070] Step 401, when receiving a test data request corresponding to the development project, search for the historical data corresponding to the test data request.

[0071] Step 403, identify the sensitive data in the historical data.

[0072] Step 405, perform desensitization processing on the sensitive data according to the desensitization rule corresponding to the data type of the sensitive data to obtain application test data.

[0073] Step 407, transfer and store the application test data to the workspace.

[0074] Among them, historical data refers to the real big data, and test data for testing big data applications can be constructed based on this part of the data. For example, historical data can be personal identity information, including name, gender, contact information, ID number, and home address, etc., and historical data can be the real data stored in the database. Data desensitization refers to the transformation of certain sensitive information through desensitization rules to achieve reliable protection of sensitive privacy data. In the case of customer security data or some commercial sensitive data, without violating the system rules, the real data is transformed and provided for testing use. Personal information such as ID numbers, mobile phone numbers, card numbers, and customer numbers all need to be desensitized.

[0075] Specifically, the development management server 104 can also be connected to the database storing historical data. When receiving a test data request corresponding to a development project, it can parse the test data request and determine which data needs to be involved in the application test data required by the development side. The application test data is constructed according to the data types required by the big data application, and corresponding desensitization processing is also required to ensure that user data is not leaked. For example, if the big data application only needs the name, contact information, and home address in the identity information, the development management server 104 can extract this part of the data from the historical data and perform corresponding data desensitization processing on these data to obtain the final application test data. In one embodiment, the big data application development method of the present application is specifically used for developing government affairs big data applications, that is, the data type is "government affairs" type data. Government affairs data is transmitted using a dedicated network and has extremely high requirements for data confidentiality and security. Therefore, during the development and testing process, sensitive data can be identified and desensitized. In this embodiment, by identifying sensitive data and performing data desensitization and other processing, the data security of the test data can be effectively improved.

[0076] In one embodiment, the desensitization rules include: data replacement, data invalidation, data randomization, data offset rounding, data masking, and flexible data encoding.

[0077] Among them, data replacement means replacing the true value with a set fixed fictional value. For example, for mobile phone numbers in sensitive data, the mobile phone numbers can be uniformly replaced with 13800013800. Data invalidation means making sensitive data desensitized by truncating, encrypting, hiding, etc. the data value, so that it no longer has exploitable value. For example, replacing the true value of the address with ******. The effects achieved by data invalidation and data replacement are basically similar. Data randomization means using random data to replace the true value and maintaining the randomness of the replacement value to simulate the authenticity of the sample. For example, using randomly generated first names and last names to replace the true value. Data offset rounding means changing digital data through random shifting. For example, changing the date 2018-01-02 8:12:25 to 2018-01-02 8:00:00. Data offset rounding ensures the approximate authenticity of the range while maintaining the security of the data. Data mask masking means masking digital data through a mask. Data mask masking is a powerful tool for desensitizing partial information of account data, such as desensitizing bank card numbers or ID card numbers. Flexible coding can be executed to meet various possible desensitization rules when special desensitization rules are required. For example, replacing the true value of the contract number with fixed letters and fixed-digit numbers. In this embodiment, corresponding data desensitization is performed according to the data characteristics of various types of data in the historical data, which can effectively ensure the confidentiality and availability of the desensitized data after data desensitization.

[0078] In one embodiment, as Figure 5 shown, transferring application test data to the workspace includes:

[0079] Step 502, generating a data migration task corresponding to the application test data.

[0080] Step 504, migrating the application test data to the corresponding storage space in the workspace through the data migration task.

[0081] Specifically, the data migration task includes information such as the migration object, the address before migration, and the address after migration, etc.

[0082] In the process of applying application test data to the workspace, after the application test data is generated, this newly generated application test data is pre-stored at a specific location in the development management server 104. To import it into the workspace for application development, it can be achieved by generating a data migration task. The data migration task designates the application test data as the migration object, and then uses the cache address of the application test data as the pre-migration address and the storage space of the workspace as the post-migration address to generate a data migration task, and completes the transfer of the test data by executing the data migration task. In one embodiment, the storage space corresponding to the test environment in the workspace can be specifically used as the post-migration address for data migration. In this embodiment, the transfer of application test data through the data migration task can effectively improve the accuracy and processing efficiency of the data transfer process.

[0083] This application also provides an application scenario that applies the above-mentioned big data application development method. Specifically, the application of the big data application development method in this application scenario is as follows:

[0084] When the developer of an application needs to develop a big data application based on government affairs big data, an application development request can be sent to the development management server equipped with the big data application development method of this application, and information such as the required computing resource information, storage resource information, and role assignment information of developers can be added to the application development request. The development management server receives the application development request, and then based on the request, creates an application development project for the development of government affairs big data applications through multi-tenant technology, allocates a workspace for the application development project according to the computing resource information, storage resource information, etc., and adds corresponding identities to different developers in the application development project according to the role assignment information of the developers. The workspace specifically includes a development / test environment and a production environment, and the relationship between the two can be referred to Figure 6, The development environment and the testing environment are used for writing task codes, configuring dependency relationships, adjusting strategies, etc. At the same time, tasks can be submitted, code audits can be conducted, tasks can be released, task approvals can be carried out, etc. to the production environment. The production environment can then feedback the application test data after data desensitization to the development environment and the production environment. At the same time, task scheduling and data output operations can also be performed in the production environment. The managers of application development publish the tasks of application development in the production environment, so that other application developers can write the corresponding task codes according to the tasks. At the same time, according to the data types that need to be processed by the government affairs big data, the corresponding big data resources can be requested from the development management server. When the development management server receives this part of the test data request, it will search for the historical data corresponding to the test data request; identify the sensitive data in the historical data; desensitize the sensitive data according to the desensitization rules corresponding to the data types of the sensitive data to obtain the application test data; transfer and store the application test data to the workspace. Among them, the desensitization rules specifically include: data replacement, data invalidation, data randomization, data offset and rounding, data masking, and data flexible coding, etc. When all the development tasks are completed, the developers can build a complete government affairs big data application based on the task codes that have been written in the workspace, and conduct corresponding tests and audits on the government affairs big data application through the desensitized application test data. According to the audit results, the final available government affairs big data application can be obtained.

[0085] In one of the embodiments, the business flow chart of the present application can be referred to Figure 7 , such as Figure 7 shown. The present application specifically involves four parties: the platform administrator of application development, the project administrator of the application development project, the project developer, and the project operation and maintenance. First, the platform administrator of application development will create an application development project, then add the project administrator, and at the same time, corresponding development or operation and maintenance personnel can also be added. Then a corresponding workspace will be allocated for project development. On the side of the project developers, they will write codes according to the released application development tasks and submit the corresponding application codes. At the same time, code audits will be conducted within the development project. If anomalies are found during the audit, re - editing can be carried out. At the same time, the project administrator can also apply to the platform for the data used for application testing. The platform will find this part of the data used for testing, desensitize it, and then submit it to the project developers. After the application development is completed, the project developers can conduct application testing through the test data, and then submit the applications that pass the test to the project administrator and the project operation and maintenance side for online approval. When the approval fails, the developers need to readjust the application, and when the approval passes, these applications can be directly released.

[0086] It should be understood that although Figures 2-5The steps in the flowchart are shown in sequence according to the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figures 2-5 At least a part of the steps in Figures 2-5 may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0087] In one embodiment, as Figure 8 shown, a big data application development device is provided. This device can be a software module, a hardware module, or a combination of both to become a part of a computer device. Specifically, the device includes: a request acquisition module 801, a project construction module 803, a task publishing module 805, a data transfer and storage module 807, and an application acquisition module 809, where:

[0088] The request acquisition module 801 is used to acquire an application development request, and the application development request is used to request the development of a big data application.

[0089] The project construction module 803 is used to create an application development project through multi-tenant technology according to the application development request, and allocate a corresponding working space for the application development project.

[0090] The task publishing module 805 is used to publish the application development tasks corresponding to the development project and acquire the task codes corresponding to the application development tasks.

[0091] The data transfer and storage module 807 is used to find the historical data corresponding to the test data request when receiving the test data request, and transfer and store the application test data corresponding to the historical data to the working space.

[0092] The application acquisition module 809 is used to test the big data application composed of the task codes through the application test data, and acquire the big data application that passes the test.

[0093] In one of the embodiments, the project construction module 803 is specifically used to: acquire the resource information and role assignment information corresponding to the big data application according to the application development request; create an application development project through multi-tenant technology, allocate a corresponding working space for the application development project according to the resource information, and allocate corresponding project roles for the developers of the application development project according to the role assignment information.

[0094] In one embodiment, it further includes a code auditing module, which is used to: audit the task code according to the task requirements corresponding to the application development task, and generate and feedback the auditing information corresponding to the task code.

[0095] In one embodiment, the code auditing module is further used to: when the auditing information indicates that the audit is passed, transfer the application development task and the task code to the environment scheduling system corresponding to the development project, and the environment scheduling system is used to provide a calling channel for the task code.

[0096] In one embodiment, the data transfer and storage module 807 is specifically used to: when receiving a test data request corresponding to the development project, search for the historical data corresponding to the test data request; identify the sensitive data in the historical data; desensitize the sensitive data according to the desensitization rules corresponding to the data type of the sensitive data to obtain application test data; transfer and store the application test data to the workspace. The desensitization rules include: data replacement, data invalidation, data randomization, data offset and rounding, data masking, and data flexible coding.

[0097] In one embodiment, the data transfer and storage module 807 is further used to: generate a data migration task corresponding to the application test data; through the data migration task, migrate the application test data to the corresponding storage space in the workspace.

[0098] For the specific limitations of the big data application development device, reference can be made to the limitations on the big data application development method in the above text, which will not be elaborated here. Each module in the above big data application development device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor in the computer device in hardware form or independent of it, or stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0099] In one embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 9 shown. The computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store big data application development data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a big data application development method.

[0100] Those skilled in the art can understand,Figure 9 The structure shown is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0101] In one embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0102] In one embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0103] In one embodiment, a computer program product or a computer program is provided. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device implements the steps in the above method embodiments.

[0104] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it may include the processes of the embodiments of the above methods. Among them, any reference to the memory, storage, database or other media used in the various embodiments provided in this application may include at least one of non-volatile and volatile memories. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0105] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0106] The above embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation to the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. A method for developing big data applications, characterized in that, the method includes: Obtain an application development request for requesting the development of a big data application; According to the application development request, create an application development project through multi-tenant technology and allocate a corresponding working space to the application development project. The working space is the overall space after proportionally allocating resources to the development environment space, test environment space, and production environment space created based on the application development project; Publish the application development tasks corresponding to the development project and obtain the task codes corresponding to the application development tasks; When a test data request is received, search for the historical data corresponding to the test data request and transfer the application test data corresponding to the historical data to the working space; Audit the task code according to the task requirements corresponding to the application development task, generate and feedback the audit information corresponding to the task code; When the audit information indicates that the audit is passed, transfer the application development task and the task code into the environment scheduling system corresponding to the development project. The environment scheduling system is used to provide a call channel for the task code. The environment scheduling system includes a development environment scheduling system and a test environment scheduling system; Test the big data application composed of the task code through the application test data to obtain a big data application that passes the test; The step of when the audit information indicates that the audit is passed, transferring the application development task and the task code into the environment scheduling system corresponding to the development project includes: When the audit information indicates that the audit is passed, transfer the application development task and the task code into the development environment scheduling system corresponding to the development project; The step of testing the big data application composed of the task code through the application test data to obtain a big data application that passes the test includes: Transfer the big data application that passes the test into the test environment scheduling system corresponding to the development project, and test the big data application composed of the task code through the application test data to obtain a big data application that passes the test.

2. The method according to claim 1, characterized in that, the step of according to the application development request, creating an application development project through multi-tenant technology and allocating a corresponding working space to the application development project includes: According to the application development request, obtain the resource information and role assignment information corresponding to the big data application; Create an application development project through multi-tenant technology, allocate a corresponding working space to the application development project according to the resource information, and allocate corresponding project roles to the developers of the application development project according to the role assignment information.

3. The method according to claim 1, characterized in that, the step of when a test data request is received, searching for the historical data corresponding to the test data request and transferring the application test data corresponding to the historical data to the working space includes: When a test data request corresponding to the development project is received, search for the historical data corresponding to the test data request; Identify the sensitive data in the historical data; Desensitize the sensitive data according to the desensitization rules corresponding to the data type of the sensitive data to obtain application test data; Transfer the application test data to the workspace.

4. The method according to claim 3, wherein, the desensitization rules include: data replacement, data invalidation, data randomization, data offset rounding, data masking, and data flexible encoding.

5. The method according to claim 3, wherein, transferring the application test data to the workspace includes: generating a data migration task corresponding to the application test data; through the data migration task, migrating the application test data to the corresponding storage space in the workspace.

6. A big data application development device, wherein, the device includes: a request acquisition module, configured to acquire an application development request, where the application development request is used to request the development of a big data application; a project construction module, configured to create an application development project through multi-tenant technology according to the application development request, and allocate a corresponding workspace to the application development project, where the workspace is an overall space after proportionally allocating resources to the development environment space, test environment space, and production environment space created based on the application development project; a task publishing module, configured to publish an application development task corresponding to the development project, and acquire a task code corresponding to the application development task; a data transfer module, configured to, when receiving a test data request, search for historical data corresponding to the test data request, and transfer the application test data corresponding to the historical data to the workspace; a code auditing module, configured to: audit the task code according to the task requirements corresponding to the application development task, generate and feedback audit information corresponding to the task code; when the audit information indicates that the audit is passed, transfer the application development task and the task code to the environment scheduling system corresponding to the development project, where the environment scheduling system is used to provide a calling channel for the task code, and the environment scheduling system includes a development environment scheduling system and a test environment scheduling system; an application acquisition module, configured to test the big data application composed of the task code through the application test data, and acquire the big data application that passes the test; the code auditing module is specifically configured to: when the audit information indicates that the audit is passed, transfer the application development task and the task code to the development environment scheduling system corresponding to the development project; the application acquisition module is specifically configured to: transfer the big data application that passes the test to the test environment scheduling system corresponding to the development project, and test the big data application composed of the task code through the application test data, and acquire the big data application that passes the test.

7. The device according to claim 6, wherein, The project construction module is specifically configured to: obtain the resource information and role assignment information corresponding to the big data application according to the application development request; create an application development project through multi-tenant technology, allocate a corresponding workspace for the application development project according to the resource information, and allocate corresponding project roles to the developers of the application development project according to the role assignment information.

8. The apparatus according to claim 6, wherein, the data transfer and storage module is specifically configured to: when receiving a test data request corresponding to the development project, search for historical data corresponding to the test data request; identify sensitive data in the historical data; perform desensitization processing on the sensitive data according to the desensitization rule corresponding to the data type of the sensitive data to obtain application test data; transfer and store the application test data to the workspace.

9. The apparatus according to claim 8, wherein, the desensitization rules include: data replacement, data invalidation, data randomization, data offset and rounding, data masking, and data flexible coding.

10. The apparatus according to claim 8, wherein, the data transfer and storage module is further configured to: generate a data migration task corresponding to the application test data; and migrate the application test data to a corresponding storage space in the workspace through the data migration task.

11. A computer device, comprising a memory and a processor, the memory storing a computer program, wherein, when the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

12. A computer-readable storage medium, storing a computer program, wherein, when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images