Script execution method, Python tool system, equipment, medium and product
Through container cloud interface interaction, pulling and automating Python tool scripts, the problem of inefficient data statistical analysis in the existing technology is solved, and efficient data analysis processing and resource management is achieved.
Patent Information
- Application Number
- CN202411399254.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-09
- Publication Date
- 2025-02-11
AI Technical Summary
In the prior art, the method of manually deriveing data and performing statistical analysis of data through hardware resources is inefficient, the human resources and time cost are high, and some statistical algorithms involve complex calculations or statistical formulas, making it difficult to achieve the target accurately and efficiently.
Through the container cloud interface interaction, scripts of the Python application environment, target Python tools and target Python tools that are to be executed, and automatically run tool scripts through target Python tools under the isolation of the running environment.
Resource reuse and resource isolation are realized, operating conflicts are reduced, Python tools are improved online hosting and task one-click automated execution capabilities, data analysis and processing efficiency are improved, and labor and time costs are reduced.
Smart Images
Figure CN119415226A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of cloud data processing, and in particular to a script execution method, a Python tool system, a device, a medium and a product. Background Art
[0002] The existing data acquisition and data processing methods mainly use manual data export and then use a variety of office software tools for statistical analysis. Although the existing data acquisition and data processing methods can achieve the purpose of self-service data acquisition and statistical analysis by using a small amount of hardware resources, they have certain limitations for complex data processing. The human resources and time costs are large, and some statistical algorithms involve complex calculations or statistical formulas, which makes it difficult to achieve the goal accurately and efficiently. Summary of the invention
[0003] The present invention provides a script execution method, a Python tool system, a device, a medium and a product, which are used to solve the problem of low efficiency in the prior art of manually exporting data and performing data statistical analysis through hardware resources.
[0004] In a first aspect, the present invention provides a script execution method, comprising: In response to a script execution request of a data processing task to be executed, calling a container cloud interface to pull the Python application environment of the data processing task to be executed; Calling a container cloud interface to pull a target Python tool and a script of the target Python tool; the target Python tool is a Python tool applied to the Python application environment; The target Python tool is called to execute the script.
[0005] In one embodiment, when calling the container cloud interface to pull the target Python tool and the script of the target Python tool, the following steps are performed for the script of the target Python tool: Call the container cloud interface to use the target object storage service OSS key and target bucket information to pull the OSS file from the OSS and obtain the hash value of the OSS file; Calling the container cloud interface to verify the integrity of the OSS file based on the hash value of the OSS file; If the integrity check of the OSS file succeeds, the container cloud interface is called to obtain the script of the target Python tool in the OSS file.
[0006] In one embodiment, calling the target Python tool to execute the script includes: Call the target Python tool based on the txt file of the script to update the Python application environment; Call the target Python tool to obtain the first file path uploaded by the user from the OSS file, and obtain the second file path of the result output from the script; Calling the target Python tool to execute the target Python tool main file, obtaining the files on the first file path, and processing the files on the first file path to obtain a result file; Calling the target Python tool to output the result file to the second file path; The target Python tool is called to push the result file on the second file path to OSS.
[0007] In one embodiment, a resource management tool is called to segment the compressed package to obtain multiple segmented compressed packages; the compressed packages are generated by the Python tool to be uploaded; Call the resource management tool to calculate the hash value of each segmented compressed package; Invoke a resource management tool to upload the plurality of segmented compressed packages and the hash value of each segmented compressed package to a backend scheduler; Calling a resource management tool to verify the integrity of each segmented compressed package uploaded to the backend scheduler based on the hash value of each segmented compressed package; If the integrity of each segmented compressed package uploaded to the backend scheduler is successfully verified, calling the resource management tool to merge the multiple segmented compressed packages uploaded to the backend scheduler in segment order to obtain the compressed package; Calling a resource management tool to decompress the compressed package to obtain the Python file of the Python tool to be uploaded; Call the resource management tool to upload the Python file to OSS.
[0008] In one embodiment, calling a resource management tool to upload the Python file to OSS includes: Calling a resource management tool to calculate a hash value of the Python file; Call the resource management tool to use the target OSS key and target bucket information to push the Python file to the specified location of OSS; Calling a resource management tool to verify the Python file pushed to a specified location of the OSS based on the hash value of the Python file; If the Python file pushed to the designated location of the OSS is verified successfully, it is determined that the Python tool to be uploaded is uploaded successfully.
[0009] In a second aspect, the present invention further provides a Python tool system, including a Python tool front-end service, a Python tool gateway service, a Python tool back-end service, a container cloud interface, and a Python tool; The Python tool front-end service is used to build and manage interactions with the front-end; The Python tool gateway service is used to receive front-end requests and route them to the back-end; The Python tool backend service is used to send the frontend request to the container cloud interface; The container cloud interface is used to respond to the front-end request, pull the Python application environment of the data processing task, the Python tool applied to the Python application environment, and the script of the Python tool; The Python tool is used to run scripts.
[0010] In one embodiment, the Python tool front-end service includes a data processing task service page and a data processing task monitoring page: The data processing task service page is used to provide users with Python tool management services; The data processing task monitoring page is used to provide users with monitoring services for the execution process of the Python tool.
[0011] In a third aspect, the present invention provides a device, comprising an electronic device, wherein the electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the script execution methods described above are implemented.
[0012] In a fourth aspect, the present invention further provides a medium, comprising a non-transitory computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the script execution methods described above are implemented.
[0013] In a fifth aspect, the present invention also provides a product, comprising a computer program product, wherein the computer program product comprises a computer program, wherein the computer program can be stored on a non-transitory computer-readable storage medium, and wherein when the computer program is executed by the processor, the steps of any of the script execution methods described above are implemented.
[0014] The script execution method, Python tool system, device, medium and product provided by the present invention, through the form of container cloud interface interaction, pull the Python application environment of the data processing task to be executed, the target Python tool that can be used in the Python application environment and the script of the target Python tool, and then automatically run the tool script through the target Python tool under the isolation of the operating environment, thereby improving the Python tool online hosting and one-click automatic execution of tasks while achieving resource reuse, resource isolation and reducing operating conflicts, thereby improving the data analysis and processing efficiency of data application developers and front-line business operators, and reducing labor and time costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0016] Figure 1 It is a flowchart diagram of the script execution method provided by the present invention.
[0017] Figure 2 It is the deployment architecture of the Python tool system provided by the present invention.
[0018] Figure 3 It is a service architecture diagram of the Python tool system provided by the present invention.
[0019] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0021] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein.
[0022] Combine the following Figure 1-Figure 4The present invention describes the script execution method, Python tool system, device, medium and product provided by the present invention.
[0023] It should be noted that operators are gradually developing a trend of attaching importance to digitalization and digital intelligent transformation, especially in terms of business operations, and have put forward higher-level requirements for front-line business operators and operation developers. In this context, in order to improve data processing efficiency, lower the technical threshold of data analysis, and reduce the cost of tool operation and maintenance promotion, the embodiment of the present invention proposes a Python tool system based on cloud data processing technology. The script execution method proposed in the embodiment of the present invention is based on the execution of the Python tool system, which can automatically run tool scripts, ultimately improving data processing efficiency, lowering the technical threshold of data analysis, and reducing the cost of tool operation and maintenance promotion.
[0024] The embodiment of the present invention describes the script execution method by taking the Python tool system as an execution subject as an example.
[0025] Figure 1 It is a flowchart diagram of the script execution method provided by the present invention.
[0026] like Figure 1 As shown, the method includes the following: Step 101: In response to a script execution request of a data processing task to be executed, call a container cloud interface to pull the Python application environment of the data processing task to be executed; Step 102: Call the container cloud interface to pull the target Python tool and the script of the target Python tool; Step 103: Call the target Python tool to execute the script.
[0027] refer to Figure 2 , Figure 2 This is the deployment architecture of the Python tool system provided by the present invention. Figure 2 It can be seen that the Python tool system adopts a two-level architecture of the underlying unified Docker and the upper heterogeneous Python environment. First, the underlying layer uses Docker to create a unified environment. Docker containers can run the same application on different machines, and each service or application can run in an independent Docker container, which can effectively achieve environmental isolation. Secondly, although the underlying layer uses a unified Docker environment, different Python environments can be configured at the upper layer according to different needs. For example, multiple different Python versions can be run on the same infrastructure, achieving development flexibility. In other words, the underlying environment is managed uniformly through Docker, which reduces the workload of operation and maintenance. At the same time, the upper heterogeneous environment ensures the flexibility of development.
[0028] Therefore, the Python tool system adopts a two-level architecture of underlying unified Docker and upper-level heterogeneous Python environment, which can support back-end capabilities and front-end functions. Among them, the back-end capabilities can realize the automatic scheduling of Python tools using cloud-based physical resources, system services and Python tasks at two levels. The front-end functions can realize the management of Python tools and Python tasks, and the distributed data management technology involved also supports internal and external network authority separation, data sharding and diversion, and heterogeneous storage.
[0029] This deployment architecture supports the expansion of NVIDIA and CUDA containerization capabilities, can effectively realize the cloudification of Python tools, realize resource reuse and isolation, reduce operation conflicts and costs; use Kubernetes to realize two-level automatic scheduling of system services and Python tasks, and realize slicing and logical mapping management of heterogeneous hardware resources. It can provide fine-grained, on-demand data scheduling and processing services, and can also adopt Kubernetes Job, Deployment and other scheduling modes according to application characteristics, allocate container resources according to data volume and realize dynamic expansion, support dynamic expansion of central processing unit (CPU) and on-demand allocation of MEM memory, reduce manual operation risks and improve efficiency.
[0030] The Python tool system adopts a two-level architecture of underlying unified Docker and upper heterogeneous Python environment, including Python tool front-end service (csap-pytool-web), Python tool gateway service (csap-pytool-ingress), Python tool back-end service (csap-pytool-capd), container cloud interface and various Python tools. Through the interaction of Python tool front-end service, Python tool gateway service, Python tool back-end service, container cloud interface and various Python tools, data processing tasks can be automatically executed with one click.
[0031] Specifically, users select data processing tasks that meet their needs on the data processing task service page provided by the Python tool front-end service. Each data processing task needs to execute the script through a specific Python tool in a specific Python application environment. When the execution is completed, the data analysis and processing is realized.
[0032] After the user selects a data processing task to be executed, the corresponding script execution request is generated by the Python tool front-end service, and the script execution request of the data processing task to be executed is transmitted to the Python tool gateway service.
[0033] The Python tool gateway service serves as the middle layer connecting the Python tool front-end service and the Python tool back-end service. It manages the script execution requests sent by the Python tool front-end service in a unified manner and routes the script execution requests to the Python tool back-end service.
[0034] After receiving the script execution request of the data processing task, the Python tool backend service can directly interact with the container cloud interface service because it contains all the functions of the Python tool system, and inform the container cloud interface of the script execution request to execute the data processing task through the scheduler.
[0035] Furthermore, the system responds to the script execution request of the data processing task to be executed, calls the container cloud interface, and pulls the Python application environment from the image repository provided by the Python tool backend service. It should be noted that if multiple data processing tasks to be executed are involved, different Python application environments will be pulled from the image repository. The Python application environment includes but is not limited to data processing Pandas, machine learning Scikit-learn, deep learning TensorFlow and PyTorch. The embodiment of the present invention takes a data processing task to be executed as an example for illustration. Of course, the system supports simultaneous processing of multiple data processing tasks to be executed.
[0036] Furthermore, the container cloud interface is called again to pull the Python tools applied to the Python application environment.
[0037] At the same time, the container cloud interface is called to access the Object Storage Service (OSS) through the permissions of the OSS. After successfully accessing the OSS, the script of the target Python tool is pulled from the OSS to the corresponding Python application environment.
[0038] After calling the Python application environment, the target Python tool applied to the Python application environment, and the script of the target Python tool through the container cloud interface, the target Python tool automatically runs the script in the Python application environment.
[0039] It should be noted that although the existing microservice architecture can also be applied to data analysis and processing, due to the diversity of the deployment environment of the microservice architecture itself, it is impossible to quickly implement service expansion. The deployment architecture of the embodiment of the present invention can achieve operating environment isolation, overcome the diversity of microservice deployment environments, and can quickly implement service expansion. It can respond quickly to instantaneous business data surges or large traffic loads.
[0040] The script execution method provided by the present invention pulls the Python application environment of the data processing task to be executed, the target Python tool that can be used in the Python application environment, and the script of the target Python tool through the form of container cloud interface interaction, and then automatically runs the tool script through the target Python tool under the isolation of the operating environment, thereby improving the online hosting of Python tools and the one-click automatic execution of tasks on the premise of realizing resource reuse, resource isolation and reducing operating conflicts, improving the data analysis and processing efficiency of data application developers and front-line business operators, and reducing labor and time costs.
[0041] In some embodiments, when calling the container cloud interface to pull the target Python tool and the script of the target Python tool, the following steps are performed for the script of the target Python tool: Call the container cloud interface to use the target object storage service OSS key and target bucket information to pull the OSS file from the OSS and obtain the hash value of the OSS file; Calling the container cloud interface to verify the integrity of the OSS file based on the hash value of the OSS file; If the integrity check of the OSS file succeeds, the container cloud interface is called to obtain the script of the target Python tool in the OSS file.
[0042] After the user uploads data and submits the task through the data processing task service page, the system will call the container cloud interface to create a POD (Platform On Demand) of the Job type, and then run the script encapsulated in the image by specifying the task command Job Command. The specific execution process of the script is as follows.
[0043] Specifically, the container cloud interface is called to use the target OSS key to verify the identity in OSS, ensuring that only authorized users can access files stored in OSS.
[0044] When accessing OSS, the location of the OSS file to be pulled is determined through the target bucket information. Based on the location of the OSS file, the corresponding OSS file is pulled and the hash value of the OSS file is obtained through the OSS application programming interface.
[0045] Further, the container cloud interface is called to calculate the hash value of the pulled OSS file. Further, the hash value calculated before the OSS file is pulled is compared with the hash value calculated after the OSS file is pulled to verify the integrity of the OSS file.
[0046] If the integrity check of the OSS file succeeds, the container cloud interface is called to obtain the script of the target Python tool in the OSS file.
[0047] The embodiment of the present invention first securely obtains the required OSS file from the OSS by using the target OSS key and target bucket information, and calculates its hash value to ensure the integrity of the file. Then, the OSS file is verified based on the hash value to ensure that the data has not been tampered with or damaged. If the integrity verification is successful, the script of the target Python tool is extracted from the OSS file. This process not only ensures the security and reliability of data transmission, but also improves the efficiency of obtaining and executing Python tool scripts, ensuring the smooth progress of subsequent operations.
[0048] In the above steps, calling the target Python tool to execute the script includes: Call the target Python tool based on the txt file of the script to update the Python application environment; Call the target Python tool to obtain the first file path uploaded by the user from the OSS file, and obtain the second file path of the result output from the script; Calling the target Python tool to execute the target Python tool main file, obtaining the files on the first file path, and processing the files on the first file path to obtain a result file; Calling the target Python tool to output the result file to the second file path; The target Python tool is called to push the result file on the second file path to OSS.
[0049] Specifically, call the target PHP tool to execute the "pip install" command, specify the "requirements.txt" file in the script, and update the Python application environment.
[0050] It should be noted that considering that many libraries have been optimized for performance during updates, updating the Python dependency environment can ensure that the application runs in the best condition.
[0051] Furthermore, the target Python tool is called to obtain the first file path uploaded by the user from the OSS file, which can be recorded as "arg_input", and the second file path of the result output is obtained from the script, which can be recorded as "arg_output". The first file path "arg_input" and the second file path "arg_output" are passed as parameters to the target Python tool main file set by the tool developer.
[0052] Further, the target Python tool is called to execute the main() function through the target Python tool main file, the file of the first file path "arg_input" is obtained, and the file of the first file path "arg_input" obtained is parsed and processed to obtain a result file.
[0053] Further, the target Python tool is called to output the result file to the second file path "arg_output".
[0054] Furthermore, the target Python tool is called to push the result file on the second file path to OSS, and OSS verifies the integrity of the result file. If the integrity of the result file is successfully verified, the container cloud interface is called to delete the Job, and the container computing resources are released.
[0055] The embodiment of the present invention updates the Python application environment to ensure the latest status of dependencies and configurations, further obtains the first file path uploaded by the user from the OSS file, and sets the second file path for the result output according to the path specified by the script, and further the target Python tool executes its main file, processes the files on the first file path, generates a result file, and finally outputs the processed result file to the second file path, and pushes it back to OSS through the target Python tool. This process not only realizes the automatic processing and storage of data and improves work efficiency, but also ensures the traceability of data.
[0056] Furthermore, the embodiment of the present invention also provides the following method: Calling a resource management tool to segment the compressed package to obtain multiple segmented compressed packages; the compressed packages are generated by the Python tool to be uploaded; Call the resource management tool to calculate the hash value of each segmented compressed package; Invoke a resource management tool to upload the plurality of segmented compressed packages and the hash value of each segmented compressed package to a backend scheduler; Calling a resource management tool to verify the integrity of each segmented compressed package uploaded to the backend scheduler based on the hash value of each segmented compressed package; If the integrity of each segmented compressed package uploaded to the backend scheduler is successfully verified, calling the resource management tool to merge the multiple segmented compressed packages uploaded to the backend scheduler in segment order to obtain the compressed package; Calling a resource management tool to decompress the compressed package to obtain the Python file of the Python tool to be uploaded; Call the resource management tool to upload the Python file to OSS.
[0057] Adding tools, modifying tools, and removing tools involve tool lifecycle management, that is, tool storage state transition. Therefore, adding or modifying tools involves tool upload operations. The specific tool upload process is as follows.
[0058] Specifically, when a tool is added or modified, the Python tool to be uploaded will generate a corresponding uploadable compressed package.
[0059] Furthermore, the resource management tool is called to segment the compressed package to obtain multiple segmented compressed packages, and the name of each segmented compressed package is set at the same time. The setting method of each segmented compressed package name needs to be in the order of the segmentation, and the corresponding serial number is set at the end of the name. That is to say, the name of each segmented compressed package will finally contain a serial number for identifying the file segmentation order.
[0060] Furthermore, the resource management tool is called to calculate the hash value of each segmented compressed package.
[0061] Furthermore, the resource management tool is called to upload multiple segmented compressed packages and the hash value of each segmented compressed package to the backend scheduler, that is, the engineering POD, which provides a device for scheduling computing resources for the Python tool service.
[0062] Furthermore, the resource management tool is called to calculate the hash value of each segmented compressed package uploaded to the backend scheduler, and the hash value calculated before and after uploading of each segmented compressed package is compared to verify the integrity of each segmented compressed package.
[0063] Furthermore, if the integrity of each segmented compressed package uploaded to the backend scheduler is successfully verified, the resource management tool is called to merge the multiple segmented compressed packages uploaded to the backend scheduler in the segmented order. At this time, the integrity of the merged compressed package can be checked again. After the integrity of the merged compressed package is successfully verified, it is determined that the merged compressed package is the original compressed package.
[0064] Further, the resource management tool is called to decompress the compressed package to obtain the Python file of the Python tool to be uploaded; Furthermore, a resource management tool is called to upload the Python file to OSS.
[0065] The embodiment of the present invention performs segmented processing on the compressed package generated by the Python tool to be uploaded, and calculates the hash value of each segment to ensure data integrity. After uploading these segmented compressed packages and their corresponding hash values to the backend scheduler, the integrity of each segment is checked according to the hash value to ensure that the data has not been tampered with. If all segments pass the integrity check, these segmented compressed packages are merged in segment order, the original compressed package is restored, the compressed package is further decompressed, the Python file of the Python tool to be uploaded is extracted, and it is uploaded to OSS. This process ensures the secure transmission and integrity check of data, and greatly improves the efficiency and reliability of tool management and upload.
[0066] In the above steps, calling the resource management tool to upload the Python file to OSS includes: Calling a resource management tool to calculate a hash value of the Python file; Call the resource management tool to use the target OSS key and target bucket information to push the Python file to the specified location of OSS; Calling a resource management tool to verify the Python file pushed to a specified location of the OSS based on the hash value of the Python file; If the Python file pushed to the designated location of the OSS is verified successfully, it is determined that the Python tool to be uploaded is uploaded successfully.
[0067] Specifically, call the resource management tool to calculate the hash value of the Python file.
[0068] Furthermore, the resource management tool is called to use the target OSS key for authentication. After successful authentication, the target bucket information is used to determine the designated location of the OSS to which the Python file is pushed, and the Python file is pushed to the designated location of the OSS.
[0069] Furthermore, the resource management tool is called to calculate the hash value of the Python file pushed to the OSS. Furthermore, the hash value calculated before the Python file is pushed is compared with the hash value after the push to verify the integrity of the Python file.
[0070] Furthermore, if the Python file pushed to the designated location of the OSS is successfully verified, it is determined that the Python tool to be uploaded is successfully uploaded.
[0071] It should be noted that the upload process is directly transmitted to OSS for two reasons: one is to make the test process in the listing phase completely simulate the normal use process after listing; the other is because the tool container is started by the container cloud, to avoid excessive involvement of the container cloud and restrictions on data transmission and port debugging solutions.
[0072] The embodiment of the present invention first calculates the hash value of the Python file to be uploaded to ensure the uniqueness and integrity of the data, and further uses the target OSS key and bucket information to safely push the Python file to the specified location of the OSS. After the file is uploaded, an integrity check is performed based on the hash value to confirm that the file pushed to the OSS is consistent with the original file. If the check is successful, it is determined that the Python tool to be uploaded is uploaded successfully. This process not only enhances the security of the data transmission process, but also ensures the integrity of the file in the cloud storage, effectively reducing the risk of data loss or error.
[0073] All of the above methods can help front-line business operators improve data analysis and processing efficiency while also having the following advantages: 1. Lower the technical threshold for data analysis, such as remuneration settlement, card number activation matching, outbound call processing, efficiency calculation, attendance processing, customer satisfaction perspective, general Excel processing and other business scenarios, starting from actual work problems and pain points, reducing the work pressure of front-line business operators; 2. Improve processing efficiency. Through the automation of data processing tasks of front-line dispatching resources, reduce the daily repetitive operations of front-line business operators and effectively improve work efficiency; 3. Reduce operation and maintenance promotion costs and provide users with one-stop services such as tool hosting and data processing capacity sharing; 4. Break through the traditional CPU and memory resource scheduling limitations, improve resource utilization and scheduling flexibility, and be suitable for personalized and diversified data analysis needs and heterogeneous file processing scenarios; 5. Shield the complex underlying container scheduling and Python execution details, automatically execute data processing tasks throughout the entire process and read configuration files according to different business logics. Users do not need any manual intervention and only need to upload, retrieve data or observe Python execution logs, which greatly shortens the work cycle and significantly improves work efficiency.
[0074] The Python tool system provided by the present invention is described below, and the Python tool system described below and the script execution method described above can be referenced to each other. During specific operation, the script execution method described in any of the above embodiments can be executed, and this embodiment will not be described in detail.
[0075] like Figure 2 As shown, the Python tool system includes a Python tool front-end service, a Python tool gateway service, a Python tool back-end service, a container cloud interface and a Python tool; The Python tool front-end service is used to build and manage interactions with the front-end; The Python tool gateway service is used to receive front-end requests and route them to the back-end; The Python tool backend service is used to send the frontend request to the container cloud interface; The container cloud interface is used to respond to the front-end request, pull the Python application environment of the data processing task, the Python tool applied to the Python application environment, and the script of the Python tool; The Python tool is used to run scripts.
[0076] Furthermore, the Python tool front-end service includes a data processing task service page and a data processing task monitoring page: The data processing task service page is used to provide users with Python tool management services; The data processing task monitoring page is used to provide users with monitoring services for the execution process of the Python tool.
[0077] Specifically, the Python tool front-end service can be recorded as csap-pytool-web, which is integrated with the front-end of the analysis platform, used to build and manage interactions with the front-end, and can interact directly with users. The Python tool front-end service can be recorded as csap-pytool-ingress, which is used to receive front-end requests and route them to the back-end. The Python tool back-end service, which includes all the functions of the Python tool system, can interact directly with the container cloud interface service, is used to send front-end requests to the container cloud interface, and is a stateful service (for example, the upload operation of a tool may include multiple shard upload requests, which can only be sent to the same back-end service and cannot be sent randomly). The container cloud interface directly manages the Python tools created by users, which are used to respond to front-end requests, pull the Python application environment for data processing tasks, the Python tools applied to the Python application environment, and the scripts of the Python tools, so that the Python tools can realize automatic script running in the isolation of the running environment. Python tool is a tool used to run scripts.
[0078] Figure 2 The engineering POD is a computer platform that provides scheduling computing resources for Python tool front-end services, Python tool gateway services, and Python tool back-end services. The container cloud interface POD is a computer platform that provides scheduling computing resources for the container cloud interface. The tool POD provides a computer platform for scheduling computing resources for Python tools.
[0079] The Python tool system is based on heterogeneous Python processing technology of co-location container facilities, data processing technology based on cloud orchestration, and distributed data management technology. It aims to help front-line business operators solve the problem of poor data analysis and processing efficiency: first, to reduce the technical threshold for data analysis by front-line operators such as data processing, indicator calculation, and general Excel processing; second, to improve processing efficiency, automate data processing for front-line scheduling resources, and solve repetitive data processing operations; third, to reduce operation and maintenance promotion costs, provide users with one-stop services such as tool hosting and sharing, and help open up a new office model.
[0080] In addition, reference Figure 3 , Figure 3 It is a service architecture diagram of the Python tool system provided by the present invention.
[0081] like Figure 3 As shown in the figure, the service architecture diagram of the Python tool system includes the user-oriented application layer, the development and operation-oriented platform layer, and the infrastructure layer. Figure 3 The application layer corresponds to Figure 2 Python tool front-end service in, Figure 3 The platform layer corresponds to Figure 2 Python tool gateway service in, Figure 3 The facility layer corresponds to Figure 2 The Python tool backend service in , the functions involved are consistent.
[0082] Among them, the application layer includes the data processing task service page and the data processing task monitoring page. The data processing task service page mainly provides users with services such as management and execution of Python tools, and the data processing task monitoring page mainly provides users with services such as monitoring and management of the execution process of Python tools. It can be understood that the core function of the data processing task service page is management, and the core function of the data processing task monitoring page is task execution. In the data processing task service page, users can click the button of the data processing task to be executed according to their needs, and can also download and view the operation manual document of the data processing task to be executed. At the same time, the page provides a top function, and users can put the tools that they are more concerned about or commonly used at the top of the data processing task. In the data processing task monitoring page, users can view the execution status of the data processing task to be executed (such as completed, failed, in progress, waiting), and allow viewing the log of the data processing task to be executed, displaying the detailed execution status through the log, and can also manually delete or batch delete tasks.
[0083] The platform layer includes user authentication (analysis platform role) management, resource (computing and storage resources, core technology GPU resource management and sharing, etc.) management, routing (stateful request forwarding) management, scheduling (POD scheduling) management, storage (tools, data packet storage) management, and synchronization (data and task life cycle synchronization) management.
[0084] The facility layer includes hardware infrastructure and container resource pool.
[0085] The layered architecture consisting of the application layer, platform layer and facility layer can clearly divide responsibilities, improve the flexibility and scalability of the system, make development and operation more efficient, improve user experience, and enhance security and resource utilization efficiency, thereby supporting diverse technical needs and quickly responding to market changes.
[0086] Figure 4 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communications interface 520 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the script execution method, which includes: in response to a script execution request of a data processing task to be executed, calling a container cloud interface to pull the Python application environment of the data processing task to be executed; calling the container cloud interface to pull the target Python tool and the script of the target Python tool; the target Python tool is a Python tool applied to the Python application environment; calling the target Python tool to execute the script.
[0087] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0088] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the script execution method provided by the above-mentioned embodiments, and the method includes: in response to a script execution request for a data processing task to be executed, calling a container cloud interface to pull the Python application environment of the data processing task to be executed; calling a container cloud interface to pull a target Python tool and a script of the target Python tool; the target Python tool is a Python tool applied to the Python application environment; calling the target Python tool to execute the script.
[0089] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the script execution method provided by the above-mentioned embodiments, the method comprising: in response to a script execution request of a data processing task to be executed, calling a container cloud interface to pull the Python application environment of the data processing task to be executed; calling a container cloud interface to pull a target Python tool and a script of the target Python tool; the target Python tool is a Python tool applied to the Python application environment; calling the target Python tool to execute the script.
[0090] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.
[0091] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A script execution method, characterized in that: include: In response to a script execution request of a data processing task to be executed, calling a container cloud interface to pull the Python application environment of the data processing task to be executed; Calling a container cloud interface to pull a target Python tool and a script of the target Python tool; the target Python tool is a Python tool applied to the Python application environment; The target Python tool is called to execute the script.
2. The script execution method according to claim 1, characterized in that: When calling the container cloud interface to pull the target Python tool and the script of the target Python tool, the following steps are performed for the script of the target Python tool: Call the container cloud interface to use the target object storage service OSS key and target bucket information to pull the OSS file from the OSS and obtain the hash value of the OSS file; Calling the container cloud interface to verify the integrity of the OSS file based on the hash value of the OSS file; If the integrity check of the OSS file succeeds, the container cloud interface is called to obtain the script of the target Python tool in the OSS file.
3. The script execution method according to claim 2, characterized in that: The calling of the target Python tool to execute the script includes: Call the target Python tool based on the txt file of the script to update the Python application environment; Call the target Python tool to obtain the first file path uploaded by the user from the OSS file, and obtain the second file path of the result output from the script; Calling the target Python tool to execute the target Python tool main file, obtaining the files on the first file path, and processing the files on the first file path to obtain a result file; Calling the target Python tool to output the result file to the second file path; The target Python tool is called to push the result file on the second file path to OSS.
4. The script execution method according to claim 1, characterized in that: Also includes: Call the resource management tool to segment the compressed package to obtain multiple segmented compressed packages; The compressed package is generated by the Python tool to be uploaded; Call the resource management tool to calculate the hash value of each segmented compressed package; Invoke a resource management tool to upload the plurality of segmented compressed packages and the hash value of each segmented compressed package to a backend scheduler; Calling a resource management tool to verify the integrity of each segmented compressed package uploaded to the backend scheduler based on the hash value of each segmented compressed package; If the integrity of each segmented compressed package uploaded to the backend scheduler is successfully verified, calling the resource management tool to merge the multiple segmented compressed packages uploaded to the backend scheduler in segment order to obtain the compressed package; Calling a resource management tool to decompress the compressed package to obtain the Python file of the Python tool to be uploaded; Call the resource management tool to upload the Python file to OSS.
5. The script execution method according to claim 4, characterized in that: The calling of the resource management tool to upload the Python file to the OSS includes: Calling a resource management tool to calculate a hash value of the Python file; Call the resource management tool to use the target OSS key and target bucket information to push the Python file to the specified location of OSS; Calling a resource management tool to verify the Python file pushed to a specified location of the OSS based on the hash value of the Python file; If the Python file pushed to the designated location of the OSS is verified successfully, it is determined that the Python tool to be uploaded is uploaded successfully.
6. A Python tool system, characterized in that: Includes Python tool front-end service, Python tool gateway service, Python tool back-end service, container cloud interface and Python tools; The Python tool front-end service is used to build and manage interactions with the front-end; The Python tool gateway service is used to receive front-end requests and route them to the back-end; The Python tool backend service is used to send the frontend request to the container cloud interface; The container cloud interface is used to respond to the front-end request, pull the Python application environment of the data processing task, the Python tool applied to the Python application environment, and the script of the Python tool; The Python tool is used to run scripts.
7. The script execution method according to claim 6, characterized in that: The Python tool front-end service includes a data processing task service page and a data processing task monitoring page: The data processing task service page is used to provide users with Python tool management services; The data processing task monitoring page is used to provide users with monitoring services for the execution process of the Python tool.
8. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the script execution method according to any one of claims 1 to 5 are implemented.
9. A computer-readable storage medium, the computer-readable storage medium comprising a non-transitory computer-readable storage medium, having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the script execution method according to any one of claims 1 to 5 are implemented.
10. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the steps of the script execution method according to any one of claims 1 to 5 are implemented.