Data processing methods, apparatus, storage media and electronic devices

Abstract

This invention discloses a data processing method, apparatus, storage medium, and electronic device. The method includes: acquiring a target data packet and corresponding configuration parameters, wherein the target data packet is a data packet to be run, and the configuration parameters are used to determine the resources to be used to run the target data packet; replacing target placeholders in a preset orchestration file based on the configuration parameters to obtain a target orchestration file, wherein the field at the position of the target placeholder is a field to be adjusted; determining a target resource space based on the target orchestration file, wherein the target resource space is used to provide an independent running resource for the target data packet; and running the target data packet based on the target resource space to obtain the running result. This invention solves the technical problem of low data runtime efficiency in resource space in related technologies.

Description

Technical Field

[0001] This invention relates to the field of big data, and more specifically, to a data processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] The workflow for billing big data can be simply described as follows: login, payment, and other related logs from various games are collected and stored in a high-throughput distributed publish-subscribe messaging system using a lightweight transmission program for forwarding and centralizing log data. These logs are called raw business logs. Currently, in the process of processing these raw logs, all jobs are managed by a single system administrator, and the entire runtime environment runs on only one application based on the resource manager. If the resource manager master node or the system administrator becomes unavailable, all jobs in the entire cluster will be affected and unable to run normally.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] At least some embodiments of the present invention provide a data processing method, apparatus, storage medium, and electronic device to at least solve the technical problem of low data runtime efficiency in the relevant technical resource space.

[0005] According to one embodiment of the present invention, a data processing method is provided, comprising: obtaining a target data packet and configuration parameters corresponding to the target data packet, wherein the target data packet is a data packet to be run, and the configuration parameters are used to determine the resources to be used to run the target data packet; replacing target placeholders in a preset orchestration file based on the configuration parameters to obtain a target orchestration file, wherein the field at the position of the target placeholder is a field to be adjusted; determining a target resource space based on the target orchestration file, wherein the target resource space is used to provide an independent running resource for the target data packet; and running the target data packet based on the target resource space to obtain a running result.

[0006] Optionally, obtaining the configuration parameters corresponding to the target data packet includes: calling a first interface to analyze the target data packet and obtain the target computing resources, wherein the target computing resources are used to represent the resources required to compute the target data packet; determining a first parameter based on the target computing resources, wherein the first parameter is the number of first components required to compute the target computing resources in the target resource space; obtaining a second parameter, wherein the second parameter is the number of second components used to manage the target resource space; and determining configuration parameters based on the first parameter and the second parameter.

[0007] Optionally, the method further includes: obtaining the target keyword of the initial layout template, wherein the target keyword is the field to be adjusted in the initial layout template; adding a target underscore at a preset position of the target keyword to obtain a target placeholder; and adjusting the initial layout template based on the target placeholder to obtain a preset layout file.

[0008] Optionally, the target resource space and the preset resource space share a log directory. During the process of running the target data packet using the target resource space, the method further includes: collecting log files of the target resource space based on the preset resource space and the log directory; monitoring the process of running the target data packet in the target resource space based on the log files, and obtaining monitoring results, wherein the monitoring results are used to indicate whether any abnormality occurs during the process of running the target data packet in the target resource space.

[0009] Optionally, the method further includes: in response to a target instruction for a preset resource space, stopping the monitoring of the processing of the target resource space based on log files, wherein the target instruction is generated within a preset time period after the target resource space stops running.

[0010] Optionally, obtaining the target data packet and the configuration parameters corresponding to the target data packet includes: in response to a touch instruction on a first control in a first interface of the target platform, obtaining the target data packet, wherein the first interface is used to determine the target data packet from at least one data packet; in response to a touch instruction on a second control in a second interface of the target platform, uploading the configuration information corresponding to the target data packet to the target platform, wherein the second interface is used to edit the configuration information corresponding to the target data packet.

[0011] Optionally, the target data packet is processed using the target resource space to obtain the processing result, including: obtaining the first version information corresponding to the target resource space; determining the target image based on the first version information, wherein the target image is used to provide an environment for running the target resource space; and running the target data packet based on the target image and the target resource space to obtain the running result.

[0012] Optionally, obtaining the first version information corresponding to the target resource space includes: responding to a touch command on a third control in a third interface on the target platform to obtain the first version information corresponding to the target resource space, wherein the third interface is used to edit the first version information corresponding to the target resource space.

[0013] Optionally, the method further includes: in response to an upgrade instruction for the target resource space, obtaining a second interface, wherein the second interface is associated with second version information, the second version information being the version information corresponding to the upgraded target resource space; and adding the second interface to a third interface.

[0014] Optionally, before analyzing the target data packet based on the first interface to obtain the target computing resources, the method further includes: preprocessing the target data packet to obtain a preprocessing result, wherein the preprocessing result is used to indicate whether the target data packet can run normally in the target environment; if the preprocessing result indicates that the target data packet can run normally in the target resources, analyzing the target data packet based on the first interface to obtain the target computing resources.

[0015] According to one embodiment of the present invention, a data processing apparatus is also provided, the apparatus comprising: an acquisition module, configured to acquire a target data packet and configuration parameters corresponding to the target data packet, wherein the target data packet is a data packet to be run, and the configuration parameters are used to determine the resources to be used to run the target data packet; a replacement module, configured to replace a target placeholder in a preset arrangement file based on the configuration parameters to obtain a target arrangement file, wherein the field at the position of the target placeholder is a field to be adjusted; a retrieval module, configured to determine a target resource space based on the target arrangement file, wherein the target resource space is used to provide an independent running resource for the target data packet; and a processing module, configured to run the target data packet based on the target resource space to obtain a running result.

[0016] According to one embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, wherein the computer program is configured to execute the data processing method described in any of the preceding claims when run by a processor.

[0017] According to one embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the data processing method described in any of the preceding claims.

[0018] In at least some embodiments of the present invention, by first obtaining the target data packet and its corresponding configuration parameters, wherein the target data packet is the data packet to be run, and the configuration parameters are used to determine the resources to be used to run the target data packet; then, based on the configuration parameters, the target placeholder in the preset orchestration file is replaced to obtain the target orchestration file, wherein the field at the position of the target placeholder is the field to be adjusted; next, the target resource space is determined based on the target orchestration file, wherein the target resource space is used to provide an independent running resource for the target data packet; finally, the target data packet is run based on the target resource space to obtain the running result, thereby improving the running efficiency of the target data packet. It is easy to note that the target orchestration file corresponding to the target data packet can be quickly determined according to the configuration parameters and the preset orchestration file, and the target resource space corresponding to the target data packet can be determined based on the target orchestration file, so as to isolate the running process of the target data packet from the running process of other data packets through the target resource space, providing an independent running resource for the target data packet, avoiding the target data packet from being affected by other factors, thereby improving the running efficiency of the target data packet, and thus solving the problem of low running efficiency of the target data packet in related technologies. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0020] Figure 1 This is a hardware structure block diagram of a mobile terminal for a data processing method according to an embodiment of the present invention.

[0021] Figure 2 This is a flowchart of a data processing method according to one embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram showing the operating status according to one embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of a user submission interface according to one embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of an editing interface according to one embodiment of the present invention;

[0025] Figure 6 This is a schematic diagram of a startup interface display according to one embodiment of the present invention;

[0026] Figure 7 This is a schematic diagram of the overall framework according to one embodiment of the present invention;

[0027] Figure 8 This is a structural block diagram of a data processing apparatus according to one embodiment of the present invention;

[0028] Figure 9 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] According to one embodiment of the present invention, an embodiment of a data processing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] This method embodiment can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, the mobile terminal can be a smartphone (such as an Android phone, iOS phone, etc.), tablet computer, PDA, mobile Internet Device (MID), PAD, game console, and other terminal devices. Figure 1 This is a hardware structure block diagram of a mobile terminal for a data processing method according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a central processing unit (CPU), graphics processing unit (GPU), digital signal processing (DSP) chip, microprocessor (MCU), programmable logic device (FPGA), neural network processor (NPU), tensor processor (TPU), artificial intelligence (AI) type processor, etc.) and a memory 104 for storing data are also shown. Optionally, the mobile terminal may further include a transmission device 106 for communication functions, an input / output device 108, and a display device 110. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0033] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data processing method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the aforementioned data processing method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0034] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0035] The inputs in input / output device 108 can come from multiple human interface devices (HIDs). Examples include keyboards and mice, gamepads, and other dedicated game controllers (such as steering wheels, fishing rods, dance mats, and remote controls). Some HIDs, in addition to providing input functions, can also provide output functions, such as force feedback and vibration from gamepads, and audio output from controllers.

[0036] Display device 110 may be, for example, a head-up display (HUD), a touchscreen liquid crystal display (LCD), and a touch display (also referred to as a "touchscreen" or "touch display"). The LCD allows a user to interact with the user interface of the mobile terminal. In some embodiments, the mobile terminal has a graphical user interface (GUI), which allows the user to interact with the GUI by touching and / or gesturing on a touch-sensitive surface. Optional human-computer interaction functions include: creating web pages, drawing, word processing, creating electronic documents, playing games, video conferencing, instant messaging, sending and receiving emails, a call interface, playing digital video, playing digital music, and / or web browsing, etc. Executable instructions for performing the above human-computer interaction functions are configured / stored in one or more processor-executable computer program products or readable storage media.

[0037] First, the technical terms or technical jargon appearing in the embodiments of this invention are explained as follows:

[0038] Flink, also known as Apache Flink, is a distributed processing engine and framework for stateful computation on bounded and unbounded data streams.

[0039] JobManager: The administrator of the Flink system, responsible for accepting and processing Flink job requests, generating execution plans for jobs, scheduling TaskManagers to run jobs, and managing and recording the status of jobs. It can be abbreviated as JM.

[0040] TaskManager: The executor of the Flink system, it is the main component responsible for executing computational tasks. It is abbreviated as TM and is essentially the JVM, namely the Java Virtual Machine.

[0041] CheckPoint: Flink's fault-tolerant recovery mechanism allows users to define the execution frequency. It records the current running state of a job at regular intervals and stores it in the database. When a job fails and restarts, it can retrieve the stored state from the database and restore the job to its previous state.

[0042] SavePoint: Flink's job recovery mechanism is a consistent mirror of the execution state of a streaming job created based on the Flink CheckPointing mechanism. It needs to be manually executed, created, and deleted by the user. Flink CheckPoint is used for restarting a job in case of unexpected failure, while SavePoint is generally used for scheduled job restarts.

[0043] FileBeat: A lightweight delivery program for forwarding and centralizing log data. In the department, it is mainly used for log collection and transmission to Kafka for storage.

[0044] Kafka is a high-throughput distributed publish-subscribe messaging system that can be used to store raw log information and provide it for consumption by Flink.

[0045] ZooKeeper is a software project that provides open-source distributed configuration services, synchronization services, and naming registration for large-scale distributed computing. In Flink, it is responsible for storing metadata about job execution and providing metadata needed for job failure recovery. It is abbreviated as ZK.

[0046] RocksDB is an architecture engine that provides key-value storage and read / write capabilities. In Flink, it can be used to store the job's running status during Checkpoint and Savepoint processes.

[0047] Kubernetes: A portable, scalable, open-source platform for managing containerized workloads and services, facilitating declarative configuration and automation. In this article, it is used as the primary tool for Flink resource tuning.

[0048] Persistent Volume Claim (PVC): A persistent volume claim represents a user's request for storage.

[0049] PersistentVolume: A persistent volume is a block of storage in a cluster, often abbreviated as PV.

[0050] Container: A container is a portable, executable, lightweight image containing software and its dependencies.

[0051] Image: A mirror image is a saved container instance that can package a set of software required for an application to run.

[0052] HDFS: Hadoop Distributed File System, also known as the Hadoop Distributed File System, is responsible for storing binary files generated during the SavePoint and CheckPoint processes in Flink.

[0053] JFlink is a platform for Flink users that manages Flink cluster job startup and shutdown, job lifecycle control, job error troubleshooting, and program package version iteration. Through this platform, users can maintain job versions, view and adjust job status, and more.

[0054] Prometheus is a system monitoring and alerting toolkit developed as open-source software. Flink has complete monitoring and data collection programs and reporting processes, which greatly help Flink operations and maintenance.

[0055] Monitoring data: Data that reflects the real-time operating status of components, usually consisting of time, monitoring indicators, and monitoring results.

[0056] Cluster configuration: This represents the basic attributes of a Flink cluster and is a necessary foundation for the stable operation of Flink jobs. It is a configuration table customized by the user according to the resource requirements of their own jobs.

[0057] The execution plan, obtained through the Plan method in the official Flink documentation, provides a Flink job topology diagram, analysis of each Flink operator job, and other results, which can expose potential problems that Flink jobs may encounter during the startup phase.

[0058] Orchestration Files: In a Kubernetes cluster, resource management and resource object orchestration deployment are handled through declarative style files, also known as resource manifest files. This article uses orchestration files to create the necessary components required for Flink to run on Kubernetes.

[0059] Figure 2 This is a flowchart of a data processing method according to one embodiment of the present invention. A graphical user interface is provided through a terminal device. The content displayed by the graphical user interface includes a touch area, such as... Figure 2 As shown, the method includes the following steps:

[0060] Step S202: Obtain the target data packet and the configuration parameters corresponding to the target data packet. The target data packet is the data packet to be run, and the configuration parameters are used to determine the resources to be used to run the target data packet.

[0061] The target data package can be uploaded by the user to the job management platform. After uploading, the target data package can be configured to obtain configuration parameters. Optionally, the target data package may include game login, payment, and other related logs. After obtaining the target data package, it can be cleaned and filtered to remove erroneous and duplicate data, converted to a format compatible with the job management platform, aggregated and grouped according to data type, and finally stored in the database. When the target data package is needed, it can be retrieved from the database and deployed to the job management platform for execution.

[0062] The configuration parameters mentioned above may include the memory size of the job management platform, the logger printing level, etc. Optionally, configurations can be made for jobs such as JobManager and TaskManager, and the number of parallel slots supported by each TaskManager can be determined.

[0063] Step S204: Replace the target placeholder in the preset arrangement file based on the configuration parameters to obtain the target arrangement file, wherein the field at the position of the target placeholder is the field to be adjusted.

[0064] The aforementioned preset orchestration file can be a template for an orchestration file that is pre-stored on the server. The configuration file on the template will have a placeholder filled in the key fields for users to modify later.

[0065] The target placeholders mentioned above can be used to temporarily replace the positions of fields to be adjusted. When used later, the fields of the placeholders can be replaced by the parameters to ensure that the parameters are successfully applied and to generate the target arrangement file.

[0066] Since the target placeholder needs to be frequently modified by the user, and the operation of the job is closely related to this field, the target placeholder can be represented by adding underscores before and after the placeholder field for distinction. In an optional embodiment, assuming {{__NAMESPACE__}} is a template for the arrangement file, all target arrangement files can be derived from this template.

[0067] For example, `taskmanager.memory.process.size:__HEAP_SIZE_TM_M__` is an orchestration file template. Parameters prefixed and terminating with underscores are target placeholders. When a user passes the target placeholder field as a parameter via the API (Application Programming Interface), such as `__HEAP_SIZE_TM_M__: 2GB`, the API will populate the template with that parameter, effectively changing `taskmanager.memory.process.size: 2GB`, and then start the cluster. This process allows users to provide a Flink cluster that meets their job requirements. If the user doesn't provide any parameters, the API will default to using 2GB of memory as the input parameter.

[0068] Optionally, the configuration parameters in the orchestration file template may include the controller orchestration files for JobManager and TaskManager, mount disk files, domain name routing entry files (also known as Image files), Flink cluster configuration files, and cluster start-up and shutdown scripts.

[0069] Optionally, by processing the keywords to be modified using placeholders, a preset arrangement file with placeholders can be obtained, which facilitates the use of different data packets.

[0070] Step S206: Determine the target resource space based on the target orchestration file, wherein the target resource space is used to provide an independent running resource for the target data packet.

[0071] The target orchestration file can be generated from a preset orchestration file. Optionally, parameters can be filled in and modified on the placeholders in the preset orchestration file to generate the target orchestration file. The target orchestration file is crucial for the correct startup of the Flink cluster.

[0072] The aforementioned target resource space can be used to provide an independent runtime resource so that different data packets can be isolated from each other, so that when the JobManager of one job becomes unavailable, other jobs will not be affected. Optionally, the target resource space can be a Flink cluster.

[0073] In one optional implementation, the Kubernetes namespace concept can be used to generate an independent namespace for each user's job. This ensures that the JobManager and TaskManager under that namespace serve only one job. That is, when job A is submitted to the cluster, Kubernetes will allocate a resource space named N_A for job A. This space only contains the required number of TaskManagers and JMs for that job. The JMs can be a model where each job exclusively uses the N_A resource and does not interfere with other job resource spaces such as N_B and N_C. This isolates jobs, achieving a Per Job execution model, where each job has its own JobManager. In other words, the target resource space can provide an independent runtime resource, allowing different data packets running concurrently to be separated. Even if the JobManager of one job becomes unavailable, it will not affect the execution of other jobs.

[0074] Step S208: Run the target data packet based on the target resource space to obtain the running result.

[0075] In an alternative embodiment, once the independent runtime resources, i.e. the target resource space, have been determined, the target data packet can be run to obtain the runtime result. Figure 3 This is a schematic diagram showing the operating status according to one embodiment of the present invention, such as... Figure 3 As shown, when the user presses the start button, the interface will display the running status in the form of a progress bar, which can be a long bar or a fan shape.

[0076] Through the above steps, the target data packet and its corresponding configuration parameters are first obtained. The target data packet is the data packet to be run, and the configuration parameters are used to determine the resources to be used to run the target data packet. Then, the target placeholders in the preset orchestration file are replaced based on the configuration parameters to obtain the target orchestration file, where the fields at the positions of the target placeholders are the fields to be adjusted. Next, the target resource space is determined based on the target orchestration file, which provides an independent running resource for the target data packet. Finally, the target data packet is run based on the target resource space to obtain the running result, thereby improving the running efficiency of the target data packet. It is noteworthy that the target orchestration file corresponding to the target data packet can be quickly determined based on the configuration parameters and the preset orchestration file, and the target resource space corresponding to the target data packet can be determined based on the target orchestration file. This allows the running process of the target data packet to be isolated from the running process of other data packets, providing the target data packet with an independent running resource. This avoids the target data packet being affected by other factors, thereby improving the running efficiency of the target data packet and solving the problem of low running efficiency of target data packets in related technologies.

[0077] Optionally, obtaining the configuration parameters corresponding to the target data packet includes: calling a first interface to analyze the target data packet and obtain the target computing resources, wherein the target computing resources are used to represent the resources required to compute the target data packet; determining a first parameter based on the target computing resources, wherein the first parameter is the number of first components required to compute the target computing resources in the target resource space; obtaining a second parameter, wherein the second parameter is the number of second components used to manage the target resource space; and determining configuration parameters based on the first parameter and the second parameter.

[0078] The first interface mentioned above can be an API interface, which is used to analyze the target data packet to obtain the target computing resources. Optionally, after analyzing the target data packet and obtaining the target computing resources, the user's request will be sent to the API backend program to further determine the first parameter based on the target computing resources.

[0079] The first parameter mentioned above is the number of first components required to calculate the target computing resources in the target resource space. Optionally, the number of first components can be the number of TaskManagers.

[0080] The second parameter mentioned above can be the number of second components used to manage the target resource space. Optionally, the number of second components can be the number of JobManagers.

[0081] In one optional embodiment, when determining the configuration parameters, the number of TaskManagers and the number of JobManagers can be determined first, and then the number of parallel slots supported by the TaskManager, the memory size corresponding to the JobManager and TaskManager, the logger printing level, and other related configuration parameters can be determined.

[0082] Optionally, the method further includes: obtaining the target keyword of the initial layout template, wherein the target keyword is the field to be adjusted in the initial layout template; adding a target underscore at a preset position of the target keyword to obtain a target placeholder; and adjusting the initial layout template based on the target placeholder to obtain a preset layout file.

[0083] The above preset positions can be the positions before or after the field to be adjusted.

[0084] The target underline mentioned above can be an underline added at a preset position.

[0085] In one optional embodiment, target underscores can be added before and after the fields to be adjusted in the initial layout template and designated as target placeholders to obtain a preset layout file with target placeholders. Optionally, in subsequent use, the user only needs to fill in the target placeholders with keywords to replace them.

[0086] Taking TaskManager-deployment as an example, a crucial field is the replica data. The number of replicas represents the number of worker components, meaning one worker component can handle one task flow simultaneously. This field determines the number of TaskManagers, i.e., the number of main components for computing tasks. A job can only run if the number of TaskManagers is sufficient to support the maximum number of operators. In the template, the keyword {{__SLOT_TM__}} can be used as a placeholder. Once the API analyzes the maximum number of operators in the user data packet, it will replace the placeholder with that number, thus ensuring that enough TaskManagers are started in the cluster for job execution.

[0087] Optionally, the target resource space and the preset resource space share a log directory. During the process of running the target data packet using the target resource space, the method further includes: collecting log files of the target resource space based on the preset resource space and the log directory; monitoring the process of running the target data packet in the target resource space based on the log files, and obtaining monitoring results, wherein the monitoring results are used to indicate whether any abnormality occurs during the process of running the target data packet in the target resource space.

[0088] The aforementioned preset resource space can share a log directory with the target resource space, thereby enabling monitoring of the target resource space. Optionally, the preset resource space can be a Filebeat cluster.

[0089] In one optional embodiment, when collecting Flink logs, two Kubernetes containers, Flink and Filebeat, can be processed using a shared Pod mode. Pod mode is a shared mode where Filebeat and Flink can share a directory, making their log paths visible to each other. Flink logs can be written to the log directory, while Filebeat collects log files in the log directory in real time. Filebeat can be used as a lightweight transfer program for forwarding and centralizing log data, primarily used in departments for log collection and transmission to Kafka for storage to ensure complete log collection.

[0090] In another optional embodiment, a monitoring data acquisition program (also known as an exporter) supported by Flink can be used to monitor the component. After enabling the exporter in the configuration, the system will automatically collect various states of the Java Virtual Machine (JVM) during Flink's operation and report them to a preset address. The preset address can be set by the user. Prometheus within the same Kubernetes cluster will recognize the preset address reported by the exporter, retrieve the data based on the preset address, and perform storage and aggregation. Finally, the operations and maintenance personnel will plot the data on an open-source data visualization tool (also known as Grafana). Optionally, monitoring data is reported to a preset address based on the monitoring status, and the monitoring data is processed based on the preset address to obtain a target table corresponding to the monitoring data.

[0091] Optionally, the method further includes: in response to a target instruction for a preset resource space, stopping the monitoring of the processing of the target resource space based on log files, wherein the target instruction is generated within a preset time period after the target resource space stops running.

[0092] The aforementioned target instruction can be a computer instruction generated after a preset time period following the shutdown of the target resource space, wherein the preset time can be specified by the user. Optionally, the target instruction can act on the target resource space, and in response to the target instruction, monitoring performed on the target resource space based on log files can be stopped.

[0093] In an alternative embodiment, to ensure the integrity of log collection, the following directive can be added to Filebeat:

[0094]

[0095]

[0096] Furthermore, this instruction allows Filebeat to only stop collecting data after Flink stops running, thus ensuring that logs are collected completely.

[0097] Optionally, obtaining the target data packet and the configuration parameters corresponding to the target data packet includes: in response to a touch instruction on a first control in a first interface of the target platform, obtaining the target data packet, wherein the first interface is used to determine the target data packet from at least one data packet; in response to a touch instruction on a second control in a second interface of the target platform, uploading the configuration information corresponding to the target data packet to the target platform, wherein the second interface is used to edit the configuration information corresponding to the target data packet.

[0098] The aforementioned target platform can be geared towards Flink users, managing Flink cluster job startup and shutdown, job lifecycle control, job error troubleshooting, and program runtime package version iteration. Users can use this platform to maintain job versions, view and adjust job status, etc. Optionally, the target platform can be the JFlink platform.

[0099] The aforementioned first control can be a control in the first interface used to obtain the target data packet. Optionally, when the user needs to obtain the target data packet, the user can perform a touch operation on the control to issue a touch command, and the target data packet can be obtained based on the touch command.

[0100] The aforementioned second control can be a control in the second interface used to upload the configuration information corresponding to the target data packet to the target platform. Optionally, when the user needs to upload the configuration information corresponding to the target data packet to the target platform, the user can perform a touch operation on the control to issue a touch command, and the configuration information corresponding to the target data packet can be uploaded to the target platform based on the touch command.

[0101] In one alternative embodiment, the JFlink platform can manage data packets through job templates. Since data packets undergo frequent version iterations, multiple versions of the data packet can coexist within a single job template. When a user selects a job template to start a job, they can view all versions of the data packet. By clicking "Upload," they can upload the data packet to the data packet list under that job template and then launch the corresponding version of the data packet according to their needs, thus completing the upload of the data packet to the target platform.

[0102] In another optional embodiment, the job name, Flink job entry point, responsible person, startup parameters, and scheduled savepoints can be configured to enable job editing. The scheduled savepoints can be used to revert to a previous state if an error is found after executing the previous statement. Optionally, when there are several versions of the background job template, each Flink version is different. The job should start the cluster by default according to the required version. Its main function is to determine the responsible person for the target version, primarily used to receive alerts and send alerts when there are problems with Kubernetes.

[0103] Figure 4 This is a schematic diagram of a user submission interface according to one embodiment of the present invention, such as... Figure 4 As shown, on the upload package interface, users need to fill in the template name, identification information, and project name. After filling in the information, the data package can be uploaded. Users can click the upload button, select "no suffix" for the version suffix, and fill in the version number. After ensuring that all information is correct, users can click "OK". If there are any errors, users can click "Cancel" and re-enter the correct information.

[0104] Optionally, the target data packet is processed using the target resource space to obtain the processing result, including: obtaining the first version information corresponding to the target resource space; determining the target image based on the first version information, wherein the target image is used to provide an environment for running the target resource space; and running the target data packet based on the target image and the target resource space to obtain the running result.

[0105] The aforementioned first version information can be the version information of the target resource space before the upgrade, where the version information can be the version number, version name, etc.

[0106] The target image described above can be used to provide the environment for running the target resource space. Based on the target image, it can be determined which version of the Flink image Kubernetes will use to start the Flink cluster. Optionally, if the image version is missing, operations personnel can quickly package and create a new one. Without special environmental requirements, the specified image can be directly pulled from the image repository or updated using the `docker pull flink` command. If there are special environmental requirements, the following steps can be followed:

[0107] Step 1: docker pull flink:version.

[0108] Fetch the latest Flink version, with the version number specified by the user.

[0109] Step 2: docker images|grep flink.

[0110] Check the image ID that was just pulled.

[0111] Step 3: docker run -it imageID / bin / bash.

[0112] Once inside the image, you can verify the next version through the application and maintain the running status. After entering the image, you can install the necessary basic software and create the environment according to the user's needs.

[0113] Step 4: Use `docker ps` to get the container ID.

[0114] Once the container ID is obtained, operations such as copying files can be performed.

[0115] Step 5: Copy the file.

[0116] docker cp, data container id: / opt / flink / lib / .

[0117] Step 6: Log in to the mirror center.

[0118] docker login URL.

[0119] Step 7: Mark the mirror image.

[0120] docker tag flink:version url:version.

[0121] Step 8: Push to the mirror center.

[0122] docker push url:version.

[0123] Furthermore, once a complete image is created, it can be used by all users, achieving a one-time solution.

[0124] Figure 5 This is a schematic diagram of an editing interface according to one embodiment of the present invention, such as... Figure 5 As shown, in the new job creation interface, users can select a job template, fill in the job name (limited to 100 characters), and choose the default startup cluster, specify the entry point, and the primary responsible person. The primary responsible person is used to receive alerts and issue alerts when Kubernetes encounters problems. Users also need to describe the job, fill in startup parameters, and select savepoint configuration. Savepoint configuration can be done arbitrarily, hourly, semi-annually, or daily. Optionally, there are several versions of the background job template, each different for a Flink version. The job must start the cluster by default according to the required version. The target version can be determined by editing the configuration in the job creation interface.

[0125] Optionally, obtaining the first version information corresponding to the target resource space includes: responding to a touch command on a third control in a third interface on the target platform to obtain the first version information corresponding to the target resource space, wherein the third interface is used to edit the first version information corresponding to the target resource space.

[0126] The aforementioned third control can be a control on a third interface. The third control can issue touch commands, and in response to the touch commands, the system will obtain the first version information corresponding to the target resource space.

[0127] In an optional embodiment, the user can manually select the data package version for starting the job, the cluster to start, and manually configure the cluster configuration required to run the job, where the parameters are all reserved fields in the orchestration file.

[0128] Figure 6 This is a schematic diagram of a startup interface display according to one embodiment of the present invention, such as... Figure 6 As shown, on the job startup interface, you can first select the job template, the current version, the startup version, the startup cluster, and the resource policy. After selecting the above information, you also need to fill in the number of task managers, the number of slots in each task manager, the memory of the task manager, and the memory of the model. Then you can click OK. If the information is incorrect, you can click Cancel and then fill it in again.

[0129] Optionally, the method further includes: in response to an upgrade instruction for the target resource space, obtaining a second interface, wherein the second interface is associated with second version information, the second version information being the version information corresponding to the upgraded target resource space; and adding the second interface to a third interface.

[0130] The aforementioned second version information can be the version information of the target resource space after the upgrade, where the version information can include version number, version name, etc.

[0131] The second interface mentioned above can be a newly built interface during the upgrade process.

[0132] In one optional embodiment, during the upgrade of the Kubernetes cluster, a higher version of Kubernetes can be set up. In this case, only a Java application programming interface (API) service needs to be set up on that cluster. JFLink uses the relationship between the data version and API stored in the database. When a new version is added, the corresponding version number and the API address corresponding to that version are inserted into the database. For example, to add Flink version 1.14.3, the command would be `insert into table(flink, v1.14.3, API)`. Users can view all the Flink versions currently stored in the database on the cluster startup interface. After selecting the desired version, the correct Flink version cluster can be started according to the corresponding API. Through this version switching, users can easily complete job migration.

[0133] The above deployment mode makes operation convenient and flexible, allowing users to choose the configuration themselves, maximizing resource utilization. Furthermore, it is also very helpful for cluster migration. Given any K8S cluster, a Flink environment can be quickly built.

[0134] Optionally, before analyzing the target data packet based on the first interface to obtain the target computing resources, the method further includes: preprocessing the target data packet to obtain a preprocessing result, wherein the preprocessing result is used to indicate whether the target data packet can run normally in the target environment; if the preprocessing result indicates that the target data packet can run normally in the target resources, analyzing the target data packet based on the first interface to obtain the target computing resources.

[0135] The aforementioned preprocessing can be a step performed on the target data packet before analysis based on the first interface. Preprocessing yields a result that can indicate whether the target data packet is corrupted or unable to run due to environmental incompatibility. Optionally, if the target data packet is normally available, it can be analyzed to obtain the target computing resources.

[0136] In one optional embodiment, after obtaining the target data packet and its corresponding configuration parameters, the job management platform can receive a data processing request from the user. This request can be sent to the API backend program. In response to the data processing request, the target data packet can be sent to the Flink client for preprocessing. Optionally, this operation can analyze the user's target data packet to obtain the maximum number of TaskManagers required for the target data packet to run, and can also expose problems such as corrupted target data packets or incompatible environments that prevent operation. Finally, the target data packets that can run normally after analysis are written into the TaskManager and JobManager by the API backend program. Using these orchestration files, a request is sent to the cluster brain (i.e., the manager in the cluster, which does not participate in execution, also known as the K8S master), which can then start a Flink on K8S cluster that meets the user's job execution needs.

[0137] In another optional embodiment, after a user submits a job, the user request can render parameters into a Kubernetes orchestration file template via API and start the cluster using the script template. Once the cluster starts, the JobManager's management address is mapped to a domain name and job ID through the ingress configuration file, thus exposing the management address. After this configuration is enabled, the cluster's JobManager interface becomes accessible to users without being isolated by the Kubernetes virtual IP environment. Once the JFlink platform obtains the JobManager's management address, it can upload the user's data packets to the cluster via the API and then use the API to start the data packets.

[0138] Figure 7This is a schematic diagram of the overall framework according to one embodiment of the present invention. As shown in the figure, the job management platform can start and stop the Flink cluster, start and stop jobs, modify cluster / job configurations, and perform timed savepoints and job status viewing operations. The interface can perform packet analysis, cluster configuration, and cluster start and stop operations. The Kubernetes layer can perform rendering configuration, routing domain names, resource scheduling, high availability guarantees, and system log operations. On the Flink cluster, job start and stop, high availability guarantees, PerJob mode, data processing, and database read and write operations can be performed. The content performed on the Kubernetes layer and the Flink cluster can be referred to as logging. When data is accessed from the database, Elasticsearch data, MySQL data, TiDB data, RocksDB data, and Kafka data can be retrieved, and notifications can be sent. The interface, Kubernetes layer, Flink cluster, and database can all perform monitoring and alarm operations. Users can also perform access control on the job management platform, interface, Kubernetes layer, Flink cluster, and database. The operating environment for all the above information can be a Kubernetes server or a standalone server.

[0139] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0140] This embodiment also provides a data processing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the terms "unit" and "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0141] Figure 8 This is a structural block diagram of a data processing apparatus according to one embodiment of the present invention, such as... Figure 8 As shown, the device includes:

[0142] The acquisition module 802 is used to acquire the target data packet and the corresponding configuration parameters of the target data packet. The target data packet is the data packet to be run, and the configuration parameters are used to determine the resources to be used to run the target data packet.

[0143] Replacement module 804 is used to replace target placeholders in a preset arrangement file based on configuration parameters to obtain a target arrangement file, wherein the field at the position of the target placeholder is the field to be adjusted.

[0144] The retrieval module 806 is used to determine the target resource space based on the target orchestration file, wherein the target resource space is used to provide an independent running resource for the target data packet.

[0145] Processing module 808 is used to run the target data packet based on the target resource space and obtain the running results.

[0146] Optionally, the acquisition module 802 includes: a calling unit, configured to call a first interface to analyze the target data packet and obtain target computing resources, wherein the target computing resources represent the resources required to compute the target data packet; a first determining unit, configured to determine a first parameter based on the target computing resources, wherein the first parameter is the number of first components required to compute the target computing resources in the target resource space; a first acquisition unit, configured to acquire a second parameter, wherein the second parameter is the number of second components used to manage the target resource space; and a second determining unit, configured to determine configuration parameters based on the first parameter and the second parameter.

[0147] Optionally, the device further includes: a second acquisition module, used to acquire target keywords of the initial layout template, wherein the target keywords are fields to be adjusted in the initial layout template; add target underscores at preset positions of the target keywords to obtain target placeholders; and adjust the initial layout template based on the target placeholders to obtain a preset layout file.

[0148] Optionally, the processing module 808 includes: a collection unit for collecting log files of the target resource space based on a preset resource space and a log directory; and a monitoring unit for monitoring the process of running the target data packet in the target resource space based on the log files and obtaining monitoring results, wherein the monitoring results are used to indicate whether any abnormality occurs during the process of running the target data packet in the target resource space.

[0149] Optionally, the monitoring unit is also configured to stop monitoring the processing of the target resource space based on the log file in response to a target instruction for the preset resource space, wherein the target instruction is generated within a preset time period after the target resource space stops running.

[0150] Optionally, the acquisition module 802 further includes: a second acquisition unit, configured to acquire a target data packet in response to a touch instruction on a first control in a first interface of the target platform, wherein the first interface is configured to determine the target data packet from at least one data packet; and an upload unit, configured to upload configuration information corresponding to the target data packet to the target platform in response to a touch instruction on a second control in a second interface of the target platform, wherein the second interface is configured to edit the configuration information corresponding to the target data packet.

[0151] Optionally, the upload unit includes: an acquisition subunit for acquiring first version information corresponding to the target resource space; a determination subunit for determining the target image based on the first version information, wherein the target image is used to provide an environment for running the target resource space; and a running subunit for running the target data packet based on the target image and the target resource space to obtain the running result.

[0152] Optionally, the acquisition sub-unit is also used to respond to a touch command on a third control in a third interface of the target platform to acquire the first version information corresponding to the target resource space, wherein the third interface is used to edit the first version information corresponding to the target resource space.

[0153] Optionally, the acquisition sub-unit is also used to acquire a second interface in response to an upgrade command for the target resource space, wherein the second interface is associated with second version information, and the second version information is the version information corresponding to the upgraded target resource space; and to add the second interface to the third interface.

[0154] Optionally, the retrieval module 806 includes: a preprocessing unit for preprocessing the target data packet to obtain a preprocessing result, wherein the preprocessing result indicates whether the target data packet can run normally in the target environment; and an analysis unit for analyzing the target data packet based on a first interface to obtain the target computing resource, provided that the preprocessing result indicates that the target data packet can run normally in the target resource.

[0155] It should be noted that the above-mentioned units and modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but not limited to these: all the above-mentioned units and modules are located in the same processor; or, the above-mentioned units and modules are located in different processors in any combination.

[0156] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0157] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0158] S1, obtain the target data packet and the corresponding configuration parameters of the target data packet, wherein the target data packet is the data packet to be run, and the configuration parameters are used to determine the resources to be used to run the target data packet.

[0159] S2, based on the configuration parameters, replace the target placeholders in the preset arrangement file to obtain the target arrangement file, wherein the field at the position of the target placeholder is the field to be adjusted.

[0160] S3, determine the target resource space based on the target orchestration file, wherein the target resource space is used to provide an independent running resource for the target data packet.

[0161] S4 runs the target data packet based on the target resource space and obtains the running results.

[0162] Optionally, in this embodiment, the computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0163] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0164] Optionally, the aforementioned computer-readable storage medium is further configured to store program code for performing the following steps:

[0165] Optionally, obtaining the configuration parameters corresponding to the target data packet includes: calling a first interface to analyze the target data packet and obtain the target computing resources, wherein the target computing resources are used to represent the resources required to compute the target data packet; determining a first parameter based on the target computing resources, wherein the first parameter is the number of first components required to compute the target computing resources in the target resource space; obtaining a second parameter, wherein the second parameter is the number of second components used to manage the target resource space; and determining configuration parameters based on the first parameter and the second parameter.

[0166] Optionally, the method further includes: obtaining the target keyword of the initial layout template, wherein the target keyword is the field to be adjusted in the initial layout template; adding a target underscore at a preset position of the target keyword to obtain a target placeholder; and adjusting the initial layout template based on the target placeholder to obtain a preset layout file.

[0167] Optionally, the target resource space and the preset resource space share a log directory. During the process of running the target data packet using the target resource space, the method further includes: collecting log files of the target resource space based on the preset resource space and the log directory; monitoring the process of running the target data packet in the target resource space based on the log files, and obtaining monitoring results, wherein the monitoring results are used to indicate whether any abnormality occurs during the process of running the target data packet in the target resource space.

[0168] Optionally, the method further includes: in response to a target instruction for a preset resource space, stopping the monitoring of the processing of the target resource space based on log files, wherein the target instruction is generated within a preset time period after the target resource space stops running.

[0169] Optionally, obtaining the target data packet and the configuration parameters corresponding to the target data packet includes: in response to a touch instruction on a first control in a first interface of the target platform, obtaining the target data packet, wherein the first interface is used to determine the target data packet from at least one data packet; in response to a touch instruction on a second control in a second interface of the target platform, uploading the configuration information corresponding to the target data packet to the target platform, wherein the second interface is used to edit the configuration information corresponding to the target data packet.

[0170] Optionally, the target data packet is processed using the target resource space to obtain the processing result, including: obtaining the first version information corresponding to the target resource space; determining the target image based on the first version information, wherein the target image is used to provide an environment for running the target resource space; and running the target data packet based on the target image and the target resource space to obtain the running result.

[0171] Optionally, obtaining the first version information corresponding to the target resource space includes: responding to a touch command on a third control in a third interface on the target platform to obtain the first version information corresponding to the target resource space, wherein the third interface is used to edit the first version information corresponding to the target resource space.

[0172] Optionally, the method further includes: in response to an upgrade instruction for the target resource space, obtaining a second interface, wherein the second interface is associated with second version information, the second version information being the version information corresponding to the upgraded target resource space; and adding the second interface to a third interface.

[0173] Optionally, before analyzing the target data packet based on the first interface to obtain the target computing resources, the method further includes: preprocessing the target data packet to obtain a preprocessing result, wherein the preprocessing result is used to indicate whether the target data packet can run normally in the target environment; if the preprocessing result indicates that the target data packet can run normally in the target resources, analyzing the target data packet based on the first interface to obtain the target computing resources.

[0174] In the computer-readable medium of this embodiment, the process involves first obtaining a target data packet and its corresponding configuration parameters, where the target data packet is the data packet to be run, and the configuration parameters are used to determine the resources to be used to run the target data packet. Then, based on the configuration parameters, target placeholders in a preset orchestration file are replaced to obtain a target orchestration file, where the field at the location of the target placeholder is the field to be adjusted. Next, a target resource space is determined based on the target orchestration file, where the target resource space is used to provide an independent running resource for the target data packet. Finally, the target data packet is run based on the target resource space to obtain the running result. It is readily apparent that after replacing the target placeholders in the preset orchestration file based on the configuration parameters to obtain the target orchestration file, the target resource space can be determined based on the target orchestration file, thereby further improving the efficiency and accuracy of data processing and solving the technical problem of low data runtime efficiency in resource spaces in related technologies.

[0175] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of the present invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (such as a CD-ROM, USB flash drive, portable hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method according to the embodiments of the present invention.

[0176] In exemplary embodiments of this application, a computer-readable storage medium stores a program product capable of implementing the methods described above in this embodiment. In some possible implementations, various aspects of the embodiments of the present invention can also be implemented as a program product comprising program code, which, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this embodiment according to various exemplary embodiments of the present invention.

[0177] According to embodiments of the present invention, a program product for implementing the above-described method may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In the embodiments of the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0178] The aforementioned program product may take the form of any combination of one or more computer-readable media. Such computer-readable storage media may be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples (not exhaustive) of computer-readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0179] It should be noted that the program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0180] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0181] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0182] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0183] S1, obtain the target data packet and the corresponding configuration parameters of the target data packet, wherein the target data packet is the data packet to be run, and the configuration parameters are used to determine the resources to be used to run the target data packet.

[0184] S2, based on the configuration parameters, replace the target placeholders in the preset arrangement file to obtain the target arrangement file, wherein the field at the position of the target placeholder is the field to be adjusted.

[0185] S3, determine the target resource space based on the target orchestration file, wherein the target resource space is used to provide an independent running resource for the target data packet.

[0186] S4 runs the target data packet based on the target resource space and obtains the running results.

[0187] Optionally, the processor may also be configured to perform the following steps via a computer program: calling a first interface to analyze the target data packet to obtain target computing resources, wherein the target computing resources represent the resources required to compute the target data packet; determining a first parameter based on the target computing resources, wherein the first parameter is the number of first components required to compute the target computing resources in the target resource space; obtaining a second parameter, wherein the second parameter is the number of second components used to manage the target resource space; and determining configuration parameters based on the first parameter and the second parameter.

[0188] Optionally, the processor may also be configured to perform the following steps via a computer program: obtaining the target keyword of the initial layout template, wherein the target keyword is the field to be adjusted in the initial layout template; adding a target underscore at a preset position of the target keyword to obtain a target placeholder; and adjusting the initial layout template based on the target placeholder to obtain a preset layout file.

[0189] Optionally, the processor may also be configured to perform the following steps via a computer program: collecting log files of the target resource space based on a preset resource space and log directory; monitoring the process of running target data packets in the target resource space based on the log files, and obtaining monitoring results, wherein the monitoring results are used to indicate whether any abnormalities occur during the process of running target data packets in the target resource space.

[0190] Optionally, the processor may also be configured to perform the following steps via a computer program: in response to a target instruction for a preset resource space, stop monitoring the processing of the target resource space based on a log file, wherein the target instruction is generated after a preset time period following the cessation of operation of the target resource space.

[0191] Optionally, the processor may also be configured to perform the following steps via a computer program: in response to a touch instruction on a first control in a first interface of the target platform, acquire a target data packet, wherein the first interface is used to determine the target data packet from at least one data packet; in response to a touch instruction on a second control in a second interface of the target platform, upload configuration information corresponding to the target data packet to the target platform, wherein the second interface is used to edit the configuration information corresponding to the target data packet.

[0192] Optionally, the processor may also be configured to perform the following steps via a computer program: obtaining first version information corresponding to the target resource space; determining a target image based on the first version information, wherein the target image is used to provide an environment for running the target resource space; and running the target data packet based on the target image and the target resource space to obtain the running result.

[0193] Optionally, the processor may also be configured to perform the following steps via a computer program: in response to a touch command on a third control in a third interface of the target platform, obtain first version information corresponding to the target resource space, wherein the third interface is used to edit the first version information corresponding to the target resource space.

[0194] Optionally, the processor may also be configured to perform the following steps via a computer program: in response to an upgrade instruction for the target resource space, obtain a second interface, wherein the second interface is associated with second version information, the second version information being the version information corresponding to the upgraded target resource space; and add the second interface to a third interface.

[0195] Optionally, the processor may also be configured to perform the following steps via a computer program: preprocessing the target data packet to obtain a preprocessing result, wherein the preprocessing result is used to indicate whether the target data packet can run normally in the target environment; if the preprocessing result indicates that the target data packet can run normally in the target resources, analyzing the target data packet based on the first interface to obtain the target computing resources.

[0196] In the electronic device of this embodiment, the process involves first acquiring a target data packet and its corresponding configuration parameters. The target data packet is the data packet to be executed, and the configuration parameters determine the resources to be used to execute the target data packet. Then, the target placeholders in a preset orchestration file are replaced based on the configuration parameters to obtain a target orchestration file, where the field at the location of the target placeholder is the field to be adjusted. Next, a target resource space is determined based on the target orchestration file, which provides an independent running resource for the target data packet. Finally, the target data packet is executed based on the target resource space to obtain the execution result. It is readily apparent that after replacing the target placeholders in the preset orchestration file based on the configuration parameters to obtain the target orchestration file, the target resource space can be determined based on the target orchestration file, thereby further improving the efficiency and accuracy of data processing and solving the technical problem of low data execution efficiency in resource spaces in related technologies.

[0197] Figure 9 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Figure 9 As shown, the electronic device 900 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0198] like Figure 9As shown, the electronic device 900 is presented in the form of a general-purpose computing device. The components of the electronic device 900 may include, but are not limited to: at least one processor 910, at least one memory 920, a bus 930 connecting different system components (including memory 920 and processor 910), and a display 940.

[0199] The memory 920 stores program code that can be executed by the processor 910, causing the processor 910 to perform the steps described in the method section of the embodiments of this application according to various exemplary implementations of the present invention.

[0200] The memory 920 may include a readable medium in the form of volatile memory cells, such as random access memory (RAM) 9201 and / or cache memory 9202, and may further include read-only memory (ROM) 9203, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory.

[0201] In some instances, memory 920 may also include programs / utilities 9204 having a set (at least one) of program modules 9205, including but not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Memory 920 may further include memory remotely located relative to processor 910, which can be connected to electronic device 900 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0202] Bus 930 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, peripheral bus, graphics acceleration port, processor 910, or a local bus using any of the various bus structures.

[0203] The display 940 may be, for example, a touchscreen liquid crystal display (LCD) that allows a user to interact with the user interface of the electronic device 900.

[0204] Optionally, the electronic device 900 can also communicate with one or more external devices 900 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 900, and / or any device that enables the electronic device 900 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via the input / output (I / O) interface 950. Furthermore, the electronic device 900 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via a network adapter 960. Figure 9 As shown, network adapter 960 communicates with other modules of electronic device 900 via bus 930. It should be understood that, although... Figure 9 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0205] The aforementioned electronic device 900 may also include: a keyboard, a cursor control device (such as a mouse), an input / output interface (I / O interface), a network interface, a power supply, and / or a camera.

[0206] Those skilled in the art will understand that Figure 9 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, the electronic device 900 may also include components that are more... Figure 9 The more or fewer components shown, or having the same Figure 1 Different configurations are shown. The memory 920 can be used to store computer programs and corresponding data, such as the computer program and corresponding data corresponding to the data processing method in this embodiment of the invention. The processor 910 executes various functional applications and data processing by running the computer program stored in the memory 920, thereby implementing the aforementioned data processing method.

[0207] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0208] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0209] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0210] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0211] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0212] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0213] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the target data packet and the configuration parameters corresponding to the target data packet, wherein the target data packet is a data packet to be run, and the configuration parameters are used to determine the resources to be used to run the target data packet; The target placeholder in the preset arrangement file is replaced based on the configuration parameters to obtain the target arrangement file, wherein the field at the position of the target placeholder is the field to be adjusted. The target resource space is determined based on the target orchestration file, wherein the target resource space is used to provide an independent running resource for the target data packet; The target data packet is executed based on the target resource space to obtain the execution result; Obtain the target data packet and the configuration parameters corresponding to the target data packet, including: In response to a touch instruction on a first control in a first interface of a target platform, the target data packet is acquired, wherein the first interface is used to determine the target data packet from at least one data packet; In response to a touch command on a second control of a second interface in the target platform, configuration information corresponding to the target data package is uploaded to the target platform, wherein the second interface is used to edit the configuration information corresponding to the target data package.

2. The method according to claim 1, characterized in that, Obtain the configuration parameters corresponding to the target data packet, including: The first interface is invoked to analyze the target data packet to obtain the target computing resources, wherein the target computing resources represent the resources required to compute the target data packet; A first parameter is determined based on the target computing resources, wherein the first parameter is the number of first components required to calculate the target computing resources in the target resource space; Obtain the second parameter, wherein the second parameter is the number of second components used to manage the target resource space; The configuration parameters are determined based on the first parameter and the second parameter.

3. The method according to claim 1, characterized in that, The method further includes: Obtain the target keywords of the initial layout template, wherein the target keywords are the fields to be adjusted in the initial layout template; Add a target underscore at a preset position of the target keyword to obtain the target placeholder; The initial layout template is adjusted based on the target placeholder to obtain the preset layout file.

4. The method according to claim 1, characterized in that, The target resource space shares a log directory with the preset resource space. During the process of running the target data packet using the target resource space, the method further includes: Log files of the target resource space are collected based on the preset resource space and the log directory; The process of running the target data packet in the target resource space is monitored based on the log file to obtain monitoring results, wherein the monitoring results are used to indicate whether any abnormality occurs in the process of running the target data packet in the target resource space.

5. The method according to claim 4, characterized in that, The method further includes: In response to a target instruction for the preset resource space, monitoring of the processing of the target resource space based on the log file is stopped, wherein the target instruction is generated within a preset time period after the target resource space stops running.

6. The method according to claim 1, characterized in that, The target data packet is processed using the target resource space to obtain a processing result, including: Obtain the first version information corresponding to the target resource space; The target image is determined based on the first version information, wherein the target image is used to provide an environment for running the target resource space; The target data packet is run based on the target image and the target resource space to obtain the running result.

7. The method according to claim 6, characterized in that, Obtaining the first version information corresponding to the target resource space includes: In response to a touch command on a third control in a third interface of the target platform, the first version information corresponding to the target resource space is obtained, wherein the third interface is used to edit the first version information corresponding to the target resource space.

8. The method according to claim 7, characterized in that, The method further includes: In response to the upgrade instruction for the target resource space, a second interface is obtained, wherein the second interface is associated with second version information, and the second version information is the version information corresponding to the upgraded target resource space; Add the second interface to the third interface.

9. The method according to claim 1, characterized in that, Before analyzing the target data packet based on the first interface to obtain the target computing resources, the method further includes: The target data packet is preprocessed to obtain a preprocessing result, wherein the preprocessing result is used to indicate whether the target data packet can run normally in the target environment; If the preprocessing result indicates that the target data packet can run normally on the target resource, the target data packet is analyzed based on the first interface to obtain the target computing resource.

10. A data processing apparatus, characterized in that, The device includes: An acquisition module is used to acquire a target data packet and the configuration parameters corresponding to the target data packet, wherein the target data packet is a data packet to be run, and the configuration parameters are used to determine the resources to be used to run the target data packet; The replacement module is used to replace the target placeholder in the preset arrangement file based on the configuration parameters to obtain the target arrangement file, wherein the field at the position of the target placeholder is the field to be adjusted; The retrieval module is used to determine the target resource space based on the target orchestration file, wherein the target resource space is used to provide an independent running resource for the target data packet; The processing module is used to run the target data packet based on the target resource space and obtain the running result; The acquisition module is further configured to acquire the target data packet in response to a touch instruction on a first control in a first interface of the target platform, wherein the first interface is configured to determine the target data packet from at least one data packet; and to upload configuration information corresponding to the target data packet to the target platform in response to a touch instruction on a second control in a second interface of the target platform, wherein the second interface is configured to edit the configuration information corresponding to the target data packet.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the data processing method according to any one of claims 1 to 9 when run by a processor.

12. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the data processing method according to any one of claims 1 to 9.

Images