Cross-platform automated data batch processing method, system, device and storage medium

Abstract

The application provides a cross-platform automatic data batch processing method, system, device and storage medium, the processing method comprises: setting a secure encrypted passwordless login cluster on a local computer; creating a data batch processing instruction for the secure encrypted passwordless login cluster to provide a cluster account, create a working directory, input and output files; creating a task execution instruction for the secure encrypted passwordless login cluster to detect whether the working task is being executed; loading and executing the data batch processing instruction in the data optimization software to start the local computer and the cluster cross-platform transmission file and execute calculation. The application realizes cross-platform cross-system passwordless interactive operation of the local computer and the cluster, reduces manual operation, maximizes the use of computing resources, greatly improves the optimization design efficiency, and provides real-time feedback of the optimization effect for timely adjustment of the optimization strategy, and has optimization case generality and expandability.

Description

Technical Field

[0001] This invention belongs to the field of high-performance computing technology, and relates to a processing method and system, particularly a cross-platform automated batch data processing method, system, device and storage medium. Background Technology

[0002] High-performance computing technology can solve the problems of batch and complex numerical calculations and iterative optimization involved in engineering design. If optimization and numerical calculations are performed only on personal computers, the optimization will be time-consuming and consume a lot of local computing resources, which may affect the normal operation of other tasks. Even with multi-core parallelism, it is difficult to meet the computational and time requirements. If optimization and numerical calculations are performed only on high-performance computing clusters, they are not suitable for multi-user operation and may cause the cluster to "paralyze" due to users' unfamiliarity with the cluster system operation.

[0003] Therefore, optimization design is typically run on optimization software on a personal local computer (usually a Windows system) to facilitate real-time monitoring of optimization effects, while batch numerical calculations require remote clustering (usually Linux or CentOS systems) to complete. Due to cross-platform compatibility and batch processing issues, current technologies mostly separate optimization and numerical calculations, making it impossible for designers to monitor optimization effects in real time and hindering timely adjustments to optimization strategies.

[0004] Therefore, how to provide a cross-platform automated batch data processing method, system, device, and storage medium to solve the shortcomings of existing technologies that separate optimization and numerical calculation, making it impossible for designers to view the optimization effect in real time and make timely adjustments to optimization strategies, has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a cross-platform automated batch data processing method, system, device and storage medium to solve the problem that the prior art separates optimization and numerical calculation, which makes it impossible for designers to view the optimization effect in real time and make timely adjustments to the optimization strategy.

[0006] To achieve the above and other related objectives, the present invention provides a cross-platform automated batch data processing method, comprising: setting up a secure encrypted passwordless login cluster on a local computer; creating data batch processing instructions for the secure encrypted passwordless login cluster to provide a cluster account, create a working directory, and input / output files; creating task execution instructions for the secure encrypted passwordless login cluster to detect whether the task is being executed; and loading and executing the data batch processing instructions in data optimization software to initiate cross-platform file transfer and computation between the local computer and the cluster.

[0007] In one embodiment of the present invention, the step of setting up a secure encrypted passwordless login cluster on a local computer includes: inputting a secure encrypted passwordless command, generating a key type for cluster login and selecting an encryption method to generate a public-private key pair, and storing the public-private key pair in the user directory of the local computer.

[0008] In one embodiment of the present invention, the step of creating a data batch processing instruction for the secure encrypted passwordless login cluster to provide a cluster account, create a working directory, and input / output files includes: creating a new text document and naming the text document a batch processing file to provide a cluster account for secure encrypted passwordless login cluster, create a working directory, and input / output files; receiving the numerical calculation file to be batch processed to the cluster working directory and uploading the numerical calculation file to be batch processed as a work task to the cluster; and when the work task is downloaded to the working directory on the local computer, feeding it back to the data optimization software pre-installed on the local computer.

[0009] In one embodiment of the present invention, the numerical calculation files to be batch processed are uploaded in a fixed calculation file format.

[0010] In one embodiment of the present invention, the task execution instruction includes a Shell script for submitting computing tasks to the cluster and a PBS script for submitting tasks to the cluster; the step of creating a task execution instruction for the secure encrypted passwordless login cluster to detect whether the work task is being executed includes: creating a Shell script for submitting computing tasks to the cluster, and creating another text document, naming the other text document a sub-batch file, to provide the cluster working directory, task submission method, and task status viewing.

[0011] In one embodiment of the present invention, the step of creating a task execution instruction for the secure encrypted passwordless login cluster to detect whether the task is being executed further includes: creating a PBS script for submitting cluster tasks, and creating another text document, naming the other text document a task submission file, to set the task submission node, task name, and task working path.

[0012] In one embodiment of the present invention, the transmission file includes an input file to be batch processed, a computational example, a mesh regeneration macro script, an output file format of target performance and constraints, and a post-processed image format.

[0013] Another aspect of the present invention provides a cross-platform automated batch data processing system, comprising: a cluster setting module for setting up a secure, encrypted, passwordless login cluster on a local computer; a first creation module for creating data batch processing instructions for the secure, encrypted, passwordless login cluster, providing a cluster account, creating a working directory, and input / output files; a second creation module for creating task execution instructions for the secure, encrypted, passwordless login cluster, to detect whether the task is being executed; and a processing module for loading and executing the data batch processing instructions in data optimization software to initiate cross-platform file transfer and computation between the local computer and the cluster.

[0014] In another aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cross-platform automated batch data processing method.

[0015] A final aspect of the present invention provides an apparatus comprising: a processor and a memory; the memory for storing a computer program, and the processor for executing the computer program stored in the memory, so that the apparatus performs the cross-platform automated batch data processing method.

[0016] As described above, the cross-platform automated batch data processing method, system, device, and storage medium of the present invention have the following beneficial effects:

[0017] The cross-platform automated batch data processing method, system, device, and storage medium described in this invention enable passwordless interactive operation between local computers and clusters across platforms and systems. It can synchronize working files / folders in real time, automatically call computing nodes, monitor in real time from both locations, and provide immediate feedback on optimization results. This reduces manual operation, maximizes the utilization of computing resources, significantly improves optimization design efficiency, and provides real-time feedback on optimization effects to facilitate timely adjustments to optimization strategies. Furthermore, it has the versatility of optimization cases and scalability. Attached Figure Description

[0018] Figure 1 The diagram shown is a flowchart of one embodiment of the cross-platform automated batch data processing method of the present invention.

[0019] Figure 2 The diagram shown illustrates an implementation method for submitting a task shell script according to the present invention.

[0020] Figure 3 The diagram shows the optimized effect of the present invention.

[0021] Figure 4 The diagram shown is a schematic representation of the principle structure of the cross-platform automated batch data processing system of the present invention in one embodiment.

[0022] Component designation explanation

[0023] 1. Cross-platform automated batch data processing

[0024] system

[0025] 40 Cluster Configuration Module

[0026] 41 First Creation Module

[0027] 42 Second Creation Module

[0028] 43 Processing Module

[0029] Steps S11 to S14 Detailed Implementation

[0030] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0031] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0032] Example 1

[0033] This embodiment provides a cross-platform automated batch data processing method, characterized by including:

[0034] Configure a secure, encrypted, passwordless login cluster on the local computer;

[0035] Create a data batch processing instruction for the secure encrypted passwordless login cluster to provide cluster account, create working directory, and input / output files;

[0036] Create task execution instructions for the secure encrypted passwordless login cluster to detect whether the task is being executed.

[0037] The data batch processing instructions are loaded and executed in the data optimization software to initiate cross-platform file transfer and computation between the local computer and the cluster.

[0038] The following will describe in detail the cross-platform automated batch data processing method provided in this embodiment, with reference to the accompanying illustrations. The cross-platform automated batch data processing method described in this embodiment can be applied to fields such as aerospace and marine engineering. Please refer to... Figure 1 The diagram shows a flowchart of a cross-platform automated batch data processing method in one embodiment. Figure 1 As shown, the cross-platform automated batch data processing method specifically includes the following steps:

[0039] S11 sets up a secure, encrypted, passwordless login cluster on the local computer.

[0040] Specifically, S11 includes: inputting the secure encryption passwordless command `ssh-keygen -t rsa` to generate a key type for cluster login and select an encryption method to produce a public-private key pair, which is then stored in the user directory of the local computer. In this embodiment, the `ssh-keygen` command is used to generate, manage, and convert authentication keys. The `-t` option specifies the type of key to be generated, and `rsa` is the most widely used public-key algorithm.

[0041] For example, in C:\User\AAAA\.ssh, the file id_rsa contains the private key, and id_rsa.pub contains the public key. Then, id_rsa.pub is uploaded to the individual user's directory on the cluster, such as / home / AAAA / .ssh, and the contents of id_rsa.pub are appended to the end of the authorized_keys file. authorized_keys is a dedicated location for storing public keys in Linux or CentOS systems. As long as the public key is placed in this file and has the correct permissions, a passwordless login to the cluster can be achieved using the individual user's private key.

[0042] S12, create a data batch processing instruction for the secure encrypted passwordless login cluster to provide cluster account, create working directory, and input / output files.

[0043] Specifically, S12 includes: creating a new text document and naming the text document a batch file to provide a secure, encrypted, passwordless login cluster account, creating a working directory, and input / output files; receiving the numerical calculation files to be batch processed to the cluster working directory, and uploading the numerical calculation files to be batch processed as a work task to the cluster in a fixed calculation file format; when the work task is downloaded to the working directory on the local computer, feeding it back to the data optimization software pre-installed on the local computer.

[0044] In this embodiment, the data batch processing instructions use a batch script. The newly created text document named run.bat mainly provides the cluster account, creates the working directory, and sets input and output files. The batch script settings can be divided into the following parts:

[0045] 1) Set the input information, as detailed below: username is the user account, unix_tmp_dir is the user's cluster working directory, unix_script is the name of the shell script used by the user to submit and view tasks on the cluster, UNIX_machine is the cluster IP address, unix_run_path is the running directory for submitted tasks, and local_case_dir is the user's local computer working path. When a user changes their working directory, only the local_case_dir information needs to be modified; when switching users, only the username needs to be modified and the local_case_dir information needs to be checked.

[0046] 2) Print the input information on the screen: To ensure that the input information is set correctly, print the input information on the screen for easy checking.

[0047] 3) Upload your local working files to the cluster working directory.

[0048] 4) Submit work tasks in the cluster.

[0049] 5) Download completed cluster tasks and clean up files to free up storage space. Upload and download use the SCP command, a built-in SSH command used for secure file / folder transfer between servers. In this application, it is used to upload numerical computation files *.sim (a computational file format of the CFD software StarCCM+), *.stl, *.java, sub.sh, and job.pbs (the contents and functions of these two files will be explained below) to the user's cluster task execution directory, download completed tasks to the local computer's working directory, and provide feedback in the local computer's data optimization software, such as CAESES.

[0050] S13, create a task execution instruction for the secure encrypted passwordless login cluster to detect whether the task is being executed. In this embodiment, the task execution instruction includes a Shell script for submitting computing tasks to the cluster and a PBS script for submitting tasks to the cluster.

[0051] In this embodiment, S13 includes:

[0052] (1) Create a Shell script to submit computing tasks to the cluster, and create another text document and name the other text document a sub-batch file to provide the cluster working directory, task submission method and view task status.

[0053] Specifically, create a shell script named sub.sh for submitting cluster computing tasks. Create another text document, name it sub, and change its file extension to .sh. sub.sh is used to provide the cluster working directory, task submission method, and how to view task status, etc. Figure 2 As shown, the task submission shell script provides the working directory path, and `chmod u+x $path / job.pbs` modifies the permissions of `job.pbs` to ensure the file has executable permissions. The task is submitted using `qsub job.pbs`. `jobid="qsub job.pbs"` provides the sequence number of the submitted task, and `job_state=`qstat|grep $jobid|awk'{print$5}'` provides the real-time status of the submitted task. "Q" indicates waiting, meaning the node is occupied and other jobs are running; "R" indicates running, meaning the submitted task is already running and computation is in progress; "E" indicates an error, meaning the task settings are incorrect. When the task computation is complete, the end message "$jobid job over" is printed to the screen.

[0054] (2) Create a PBS script for submitting cluster tasks, and create another text document named Task Submission File to set the task submission node, task name and task working path.

[0055] Specifically, create a PBS script job.pbs for submitting tasks to the cluster. Create a new text document, name it job, and change the file extension to .pbs. job.pbs is used to set the submission nodes, task name, and the working path of the task. #PBS-N ship sets the task name for submission; #PBS-I nodes=cu01:ppn=16+cu02:ppn=16 indicates using nodes 01 and 02, each using 16 cores. This can be set according to the specific cluster configuration and individual computing needs. You can specify specific nodes or only specify the number of nodes to use. The former will complete the computation task on the specified nodes without affecting other user jobs, while the latter will not guarantee computation on fixed nodes; it will run as long as there are empty nodes. The choice between the two methods depends on the situation. #PBS-q batch specifies the queue name. MYINPUTFILE=“ship_resistance_prediction.sim” specifies the computation example name. The command `starccm+-np$NHOSTS-machinefile$(pwd) / $JOBID / nodefile-power-batch-rsh ssh$(pwd) / remodel.java$MYINPUTFILE>$(pwd) / result.log` invokes parallel computation on the Starccm+ cluster and outputs the computation process to the `result.log` log file in real time.

[0056] S14, the data batch processing instructions are loaded and executed in the data optimization software to initiate cross-platform file transfer and computation between the local computer and the cluster. The transferred files include the input files to be batch processed, computational examples, mesh regeneration macro scripts, target performance and constraint output file formats, and post-processed image formats, etc.

[0057] For example, in step S14, the Batch script run.bat is loaded into the data optimization software, setting up the ship deformation method and preparing the interactive settings for the data optimization software CAESES and the numerical calculation software StarCCM+, including the ship input file, calculation examples, mesh regeneration macro script, target performance and constraint output file format, and post-processing image format, etc. Cross-platform file transfer and calculation are initiated between the local computer and the cluster, at which point the calculation status and optimization effects (such as...) can be monitored. Figure 3 (as shown), until the calculation is complete.

[0058] This embodiment uses the ship hull optimization software CAESES in the shipbuilding industry as an example for cross-platform automated batch data processing, and StarCCM+ is used for numerical calculations, but it is not limited to this software or even this industry. This method is versatile; for example, it can be applied to the aerospace and marine engineering fields. It is suitable for both optimization software Isight and numerical calculations using Ansys software. Even user-developed program packages can achieve cross-platform, cross-system batch automated numerical calculations through this method or with slight modifications, promoting the application of high-performance computing technology in optimization design in the engineering field.

[0059] The described cross-platform automated batch data processing method enables passwordless interactive operation between local computers and clusters across platforms and systems. It can synchronize working files / folders in real time, automatically call computing nodes, monitor in real time from both locations, and provide immediate feedback on optimization results. This reduces manual operation, maximizes the use of computing resources, significantly improves optimization design efficiency, and provides real-time feedback on optimization effects to facilitate timely adjustments to optimization strategies. Furthermore, it has the versatility of optimization cases and scalability.

[0060] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following... Figure 1 The cross-platform automated batch data processing method.

[0061] At any possible level of technical detail, this application can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this application.

[0062] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0063] The computer-readable program described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and forwards these instructions to a computer-readable storage medium in the respective computing / processing device. The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and procedural programming languages ​​such as "C" or similar programming languages. Computer-readable program instructions may execute entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of this application.

[0064] Finally, this embodiment provides a cross-platform automated batch data processing system, including:

[0065] The cluster settings module is used to set up a secure, encrypted, passwordless login cluster on the local computer.

[0066] The first creation module is used to create data batch processing instructions for the secure encrypted passwordless login cluster, providing cluster accounts, creating working directories, and input / output files;

[0067] The second creation module is used to create task execution instructions for the secure encrypted passwordless login cluster, in order to detect whether the task is being executed.

[0068] The processing module is used to load and execute the data batch processing instructions in the data optimization software to initiate cross-platform file transfer and computation between the local computer and the cluster.

[0069] The following will describe in detail the cross-platform automated batch data processing system provided in this embodiment, with reference to the accompanying drawings. Please refer to... Figure 4 The diagram shows the principle structure of a cross-platform automated batch data processing system in one embodiment. Figure 4 As shown, the cross-platform automated batch data processing system 4 includes a cluster setting module 40, a first creation module 41, a second creation module 42, and a processing module 43.

[0070] The cluster setting module 40 is used to set up a secure, encrypted, passwordless login cluster on a local computer.

[0071] Specifically, the cluster setup module 40 inputs the secure encryption passwordless command `ssh-keygen -t rsa` to generate a key type for cluster login and select an encryption method, thus generating a public-private key pair, which is then stored in the user directory of the local computer. In this embodiment, the `ssh-keygen` command is used to generate, manage, and convert authentication keys. The `-t` option specifies the type of key to be generated, and `rsa` is the most widely used public-key algorithm.

[0072] The first creation module 41 is used to create data batch processing instructions for the secure encrypted passwordless login cluster, so as to provide cluster account, create working directory, and input / output files.

[0073] Specifically, the first creation module 41 creates a new text document and names it a batch file to provide a secure, encrypted, passwordless login cluster account, create a working directory, and input / output files. When the numerical calculation file to be batched is received and uploaded to the cluster working directory, it is uploaded to the cluster as a task in a fixed calculation file format. When the task is downloaded to the working directory on the local computer, it is fed back to the data optimization software pre-installed on the local computer.

[0074] The second creation module 42 is used to create task execution instructions for the secure encrypted passwordless login cluster to detect whether the work task is being executed. In this embodiment, the task execution instructions include a Shell script for submitting computing tasks to the cluster and a PBS script for submitting tasks to the cluster.

[0075] Specifically, the second creation module 42 creates a Shell script for submitting computing tasks to the cluster, and creates another text document, naming the other text document a sub-batch file, to provide the cluster working directory, task submission method, and task status viewing.

[0076] The second creation module 42 also creates a PBS script for submitting cluster tasks and creates another text document named the task submission file to set the task submission node, task name, and task working path.

[0077] The processing module 43 is used to load and execute the data batch processing instructions in the data optimization software to initiate cross-platform file transfer and computation between the local computer and the cluster. The transferred files include the input files to be batch processed, computational examples, mesh regeneration macro scripts, target performance and constraint output file formats, and post-processed image formats, etc.

[0078] It should be noted that the division of the various modules in the above system is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls, entirely in hardware, or partially in software calls via processing element calls, with some modules implemented in hardware. For example, module x can be a separate processing element or integrated into a chip within the system. Additionally, module x can be stored as program code in the system's memory, invoked and executed by a processing element. The implementation of other modules is similar. These modules can be fully or partially integrated together or implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the processor element or through software instructions. These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), one or more Field Programmable Gate Arrays (FPGAs), etc. When a module is implemented through processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. These modules can be integrated together to form a System-on-a-Chip (SOC).

[0079] Example 2

[0080] This embodiment provides a device, which includes: a processor, a memory, a transceiver, a communication interface and / or a system bus; the memory and the communication interface are connected to the processor and the transceiver through the system bus and complete mutual communication; the memory is used to store computer programs; the communication interface is used to communicate with other devices; and the processor and the transceiver are used to run the computer programs, so that the device performs the various steps of the cross-platform automated batch data processing method as described in Embodiment 1.

[0081] The system bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices (such as clients, read-write libraries, and read-only libraries). Memory may include Random Access Memory (RAM) and may also include non-volatile memory, such as at least one disk drive.

[0082] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0083] The scope of protection of the cross-platform automated batch data processing method described in this invention is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this invention is included within the scope of protection of this invention.

[0084] This invention also provides a cross-platform automated batch data processing system, which can implement the cross-platform automated batch data processing method described in this invention. However, the implementation device of the cross-platform automated batch data processing method described in this invention includes, but is not limited to, the structure of the cross-platform automated batch data processing system listed in this embodiment. All structural modifications and substitutions of the prior art made in accordance with the principles of this invention are included within the protection scope of this invention.

[0085] In summary, the cross-platform automated batch data processing method, system, device, and storage medium described in this invention enable passwordless interactive operation between local computers and clusters across platforms and systems. It can synchronize working files / folders in real time, automatically call computing nodes, and monitor both locations in real time, providing immediate feedback on optimization results. This reduces manual operation, maximizes the utilization of computing resources, significantly improves optimization design efficiency, and provides real-time feedback on optimization effects for timely adjustments to optimization strategies. Furthermore, it possesses universality in optimization cases and scalability. Therefore, this invention effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0086] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A cross-platform automated batch data processing method, characterized in that, include: Configure a secure, encrypted, passwordless login cluster on the local computer; Create a data batch processing instruction for the secure encrypted passwordless login cluster to provide cluster account, create working directory, and input / output files; Create task execution instructions for the secure encrypted passwordless login cluster to detect whether the task is being executed. The data batch processing instructions are loaded and executed in the data optimization software to initiate cross-platform file transfer and computation between the local computer and the cluster. The task execution instructions include a Shell script for submitting computing tasks to the cluster and a PBS script for submitting tasks to the cluster. The steps for creating task execution instructions for the secure, encrypted, passwordless login cluster, and for detecting whether the task is being executed, include: Create a Shell script to submit computing tasks to the cluster, and create a new text document named a sub-batch file to provide the cluster working directory, task submission method, and task status viewing. The step of creating task execution instructions for the secure encrypted passwordless login cluster, and detecting whether the task is being executed, further includes: Create a PBS script for submitting tasks to the cluster, and create another text document named "Task Submission File" to set the task submission node, task name, and working path of the task.

2. The cross-platform automated batch data processing method according to claim 1, characterized in that, The steps for setting up a secure, encrypted, passwordless login cluster on a local computer include: Enter the secure encryption command without a password to generate a key type for cluster login and select the encryption method to produce a public-private key pair, which is then stored in the user directory of the local computer.

3. The cross-platform automated batch data processing method according to claim 1, characterized in that, The steps for creating a data batch processing instruction for the secure, encrypted, passwordless login cluster, providing a cluster account, creating a working directory, and specifying input / output files, include: Create a new text document and name it a batch file to provide a secure, encrypted, passwordless login cluster account, create a working directory, and input / output files; Once the numerical calculation files to be batched are received and uploaded to the cluster working directory, they are uploaded to the cluster as a work task. When the work task is downloaded to the working directory on the local computer, it is fed back to the data optimization software pre-installed on the local computer.

4. The cross-platform automated batch data processing method according to claim 3, characterized in that, Upload the numerical calculation files to be processed in a fixed calculation file format.

5. The cross-platform automated batch data processing method according to claim 1, characterized in that, The transmitted files include the input files to be batch processed, computational examples, mesh regeneration macro scripts, target performance and constraint output file formats, and post-processed image formats.

6. A cross-platform automated batch data processing system, characterized in that, include: The cluster settings module is used to set up a secure, encrypted, passwordless login cluster on the local computer. The first creation module is used to create data batch processing instructions for the secure encrypted passwordless login cluster, providing cluster accounts, creating working directories, and input / output files; The second creation module is used to create task execution instructions for the secure encrypted passwordless login cluster, in order to detect whether the work task is being executed. The processing module is used to load and execute the data batch processing instructions in the data optimization software to initiate cross-platform file transfer and computation between the local computer and the cluster. The task execution instructions include a Shell script for submitting computing tasks to the cluster and a PBS script for submitting tasks to the cluster. The steps for creating task execution instructions for the secure, encrypted, passwordless login cluster, and for detecting whether the task is being executed, include: Create a Shell script to submit computing tasks to the cluster, and create a new text document named a sub-batch file to provide the cluster working directory, task submission method, and task status viewing. The step of creating task execution instructions for the secure encrypted passwordless login cluster, and detecting whether the task is being executed, further includes: Create a PBS script for submitting tasks to the cluster, and create another text document named "Task Submission File" to set the task submission node, task name, and working path of the task.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the cross-platform automated batch data processing method as described in any one of claims 1 to 5.

8. An electronic device, characterized in that, include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the cross-platform automated batch data processing method as described in any one of claims 1 to 5.

Images