Geophysical software containerization mirror image manufacturing method and storage medium

Abstract

The invention provides a geophysical software containerization mirror image manufacturing method and a storage medium, and belongs to the field of computer information technology and petroleum geophysical prospecting. The method comprises the following steps of: 1, packaging geophysical software into a mirror image; 2, performing version control on the mirror image of the geophysical software; step 3, carrying out layered construction on the mirror image of the geophysical software; step 4, performing standardized configuration on the calculation environment of the geophysical software; and step 5, fast batch deployment of geophysical software mirror images. According to the method, efficient utilization of resources and performance optimization are fully considered, and consumption of computing resources is reduced to the greatest extent while it is ensured that each simplified mirror image meets functional requirements.

Description

Technical Field

[0001] This invention belongs to the fields of computer information technology and petroleum geophysical exploration, specifically relating to the technology for creating containerized images of geophysical software. More specifically, it relates to a method for creating containerized images of geophysical software and a storage medium therefor. Background Technology

[0002] Cloud-native container image technology is one of the key technologies in the cloud-native ecosystem. A container image is a template for a container, packaging a complete operating system and application runtime environment. It consists of multiple incremental layer files, each essentially a compressed package of differences (including file additions, modifications, or deletions) from the previous layer. This way, during image distribution, only the layer files need to be distributed; the container engine can stack the differences according to a pre-defined method and mount them to a specified directory in read-only mode. However, some problems have been encountered in the development of container image technology. For example, in traditional methods, container startup is slow, especially when the image size is large, downloading the entire image takes a long time, and file data cannot be loaded on demand. Simultaneously, storing different images locally incurs high storage costs, layer sharing between images is inefficient, and image centers may contain a large number of similar images, leading to storage waste. To solve these problems, the industry has made numerous attempts, with typical technologies including overlaybd. Taking Alibaba's open-source overlaybd image format as an example, it implements network-based on-demand reading, enabling containers to start quickly. However, specifically in the field of petroleum geophysical exploration, in recent years, as the scale of the geophysical cloud computing center of the Geophysical Exploration Institute has gradually expanded, the objects it faces for operation and maintenance have become increasingly large and complex. The traditional physical machine-based geophysical software deployment model is inefficient and difficult to manage. The Geophysical Exploration Institute has significant differences in hardware configuration, hardware age and health, and hardware computing power. There are many types and versions of software, deployment models are highly coupled with the underlying hardware, migration and switching costs are high, and there is no unified scheduling and management. These current challenges have presented new demands and challenges for the effective integration of cloud-native technology with the geophysical exploration industry, especially with geophysical high-performance computing services. Summary of the Invention

[0003] The purpose of this invention is to solve the problems existing in the prior art. In view of the shortcomings of the prior art, this invention provides a method for creating containerized images of geophysical software, which aims to solve the problems of low efficiency and high management difficulty of traditional physical machine-based geophysical software deployment mode.

[0004] This invention is achieved through the following technical solution:

[0005] In a first aspect, the present invention provides a method for creating containerized images of geophysical software, the method comprising:

[0006] Step 1: Package the geophysical software into an image;

[0007] Step 2: Implement version control for the mirror of the geophysical software;

[0008] Step 3: Construct layered images from geophysical software;

[0009] Step four: Standardize the configuration of the geophysical software computing environment;

[0010] Step 5: Rapid batch deployment of geophysical software mirrors.

[0011] Furthermore, the operation in step one includes: packaging the geophysical software and its dependencies into an independent, portable image; the image is stored in an image repository.

[0012] Furthermore, the image is stored in a hierarchical file system structure, and the image is stored in a high-speed storage layer or a low-cost storage layer according to the access frequency.

[0013] Furthermore, step one also includes: using geophysical scheduling strategies to develop resource templates for jobs with different resource load levels, so that mirrors of different computing types can quickly generate computing units according to different resource requirement templates.

[0014] Furthermore, step one also includes: based on namespace and cgroups features, restricting the system calls that the geophysical high-performance computing process within the container can execute; restricting the access permissions of processes within the container to files and network resources, thereby achieving secure environment isolation between multiple computing units.

[0015] Furthermore, step two includes:

[0016] Step 21: Different versions of the same geophysical software are used to construct differential tags;

[0017] Step 22: Simplify and build mirror images from different geophysical software.

[0018] Step 23: Establish a rollback strategy to facilitate version rollback and change tracking.

[0019] Furthermore, step 21 includes: targeted tagging construction based on Dockerfile and Dockercompose technology for different geophysical software and characteristics, and different types of computing tasks.

[0020] Furthermore, step 22 includes: performing multi-stage image difference construction for different computing scenarios, different construction parameters, different construction conditions, and different base images; and selecting a smaller intermediate image for subsequent construction steps.

[0021] Furthermore, step three includes: utilizing the layered characteristics of the image to divide the image into a base layer, a dependency layer, an application layer, and a writable layer, in order to improve build efficiency and save storage space.

[0022] In a second aspect, the present invention provides a computer-readable storage medium storing at least one computer-executable program, wherein when the at least one program is executed by the computer, it enables the steps of the geophysical software containerization image creation method.

[0023] Compared with existing technologies, the beneficial effects of this invention are: it further optimizes the containerization process of geophysical software. In the process of decomposing the main functions of the software, it fully considers the efficient use of resources and performance optimization, ensuring that each simplified image meets functional requirements while minimizing computational resource consumption. Attached Figure Description

[0024] Figure 1 It is a breakdown of the geophysical software deployment architecture.

[0025] Figure 2 It is a geophysical software computing architecture.

[0026] Figure 3 This is a schematic diagram of the design and data transmission of a geophysical software mirror repository.

[0027] Figure 4 It is a dependency of the core elements of geophysical software image encapsulation.

[0028] Figure 5 It is a geophysical mirror repository and cloud computing architecture. Detailed Implementation

[0029] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments:

[0030] The primary objective of this invention is to provide a method and technology for encapsulating geophysical software and its related operating environment and dependencies into a container image. By analyzing the geophysical software architecture, computing environment, and business scenarios, the main functional components of the geophysical software are broken down, and simplified images are created according to different software functions, providing a standardized and readily usable computing environment. Simultaneously, a containerized image repository for geophysical software is developed and designed, supporting the creation of differentiated images of different versions of the same geophysical software using unique tags, simplified image creation for different geophysical software, and a multi-copy security design for the geophysical software image repository; it also supports real-time distribution and updates across large-scale computing nodes. Based on the geophysical software image repository, real-time distribution and updates of business data across public and private networks can be achieved, effectively improving operational and deployment efficiency, reducing costs, and enhancing the robustness and applicability of the entire business system.

[0031] like Figure 1 As shown, the deployment architecture of geophysical software typically includes the following main components:

[0032] (1) Hardware Basics

[0033] Servers: High-performance servers are the core hardware for deploying geophysical software. Appropriate server configurations, including processor performance, memory capacity, storage capacity, and disk read / write speeds, can be selected based on the software's requirements and user scale.

[0034] For large-scale geophysical data processing and simulation, multiple servers may be required to form a cluster to improve computing power and data processing speed.

[0035] Storage devices: Geophysical software typically processes large amounts of data, thus requiring reliable storage devices. Options include hard drive arrays, network storage devices, or cloud storage services.

[0036] The capacity and performance of storage devices should be selected based on the software's data storage requirements and access patterns, while also considering data backup and recovery strategies.

[0037] Network equipment: A stable network connection is crucial for ensuring the proper functioning of geophysical software. High-speed network switches, routers, and firewalls are required to guarantee data transmission speed and security.

[0038] For distributed geophysical software, issues such as network latency and bandwidth limitations also need to be considered to ensure that the software works collaboratively in different locations.

[0039] (2) Software Environment

[0040] Operating System: For geophysical software that requires high-performance computing, it may be necessary to choose a dedicated high-performance computing operating system, such as CentOS or Ubuntu Server.

[0041] Database Management System (DBMS): Geophysical software typically requires storing and managing large amounts of data, necessitating the selection of a suitable DBMS. Relational DBMSs such as MySQL, Oracle, and SQL Server can be chosen, as can non-relational DBMSs such as MongoDB and Redis. The selection of a DBMS should be based on the software's data storage requirements, access patterns, and performance requirements, while also considering data backup and recovery strategies.

[0042] Geophysical Software: Install and configure the geophysical software according to the installation guide. During installation, pay attention to software dependencies and configuration parameter settings to ensure the software functions correctly.

[0043] Geophysical software typically includes functional modules such as data processing, simulation, and visualization, which need to be selected and configured according to the user's needs and the software's functions.

[0044] Geophysical software is based on traditional physical hardware deployment technology. This invention innovatively upgrades geophysical software from the traditional physical hardware deployment model to a cloud-native system deployment model. Geophysical containerization image technology is a key point of the cloud deployment model, and the decomposition and abstraction of the physical hardware deployment model is a prerequisite for the creation of containerized images.

[0045] Currently, geophysical software primarily utilizes a cluster-based serial-parallel computing architecture, which combines multiple physical computers into a cluster connected via a high-speed network to collaboratively complete geophysical computation tasks. The computational architecture of geophysical software needs to be selected and designed based on factors such as the scale of the computational task, the amount of data, and the computational resource requirements. Figure 2 As a geophysical software computing architecture, this invention studies and analyzes existing software computing architectures, and uses different technical routes to containerize and encapsulate the architectural components into images, forming a method suitable for the field of geophysical high-performance computing.

[0046] The embodiments of the method of the present invention are mainly divided into the following aspects:

[0047] Example 1:

[0048] Step 1: Geophysical Software Packaging

[0049] Packaging an application and its dependencies (such as libraries and runtime environments) into a single, portable unit ensures that the software runs in the same way across different environments. For example, a geophysical software installation environment, system dependencies, and configurations are all encapsulated in a single image.

[0050] The packaged images are stored in an image repository. In this invention, the container image repository is a crucial component for storing and distributing container images. The following are the specific details of its technical implementation:

[0051] (1) Data storage

[0052] Firstly, container images are stored in a layered file system structure. The image repository utilizes object storage technology, storing each image layer as an independent object. This approach facilitates efficient management and retrieval of image layers, and avoids duplicate storage when multiple images share the same layer, thus saving storage space.

[0053] Secondly, version control involves tagging each image with a version number so that users can select a specific version for deployment. A mechanism similar to a version control system is used to record the image's change history, facilitating rollback to a specific version.

[0054] (2) Data distribution

[0055] To improve image download speed and availability, the container image repository of this invention can be integrated with a CDN. When a user requests an image, the CDN provides the image from the node closest to the user, reducing network latency; preferably, the CDN can also cache commonly used geophysical software images, further improving distribution efficiency.

[0056] This invention employs an on-demand retrieval and pre-retrieval strategy: the image is only retrieved from the repository when the user needs it; further, pre-retrieval is used to push the image to the target node in advance when it is anticipated that the user may need the image, in order to improve the response speed.

[0057] (3) Security Control

[0058] Implement strict access control policies to ensure that only authorized users can access specific images. Access control can be based on user roles, groups, or projects to restrict access to sensitive images. This invention uses multiple authentication methods, including username / password, tokens, and certificates.

[0059] Furthermore, image publishers can digitally sign their images to ensure their integrity and authenticity. Image repositories verify digital signatures when distributing images to prevent the spread of malicious images. Users can also verify signatures when pulling images to ensure they come from a trusted source.

[0060] (4) Interfaces and Protocols

[0061] This invention adopts a container image specification with geophysical business characteristics, follows open container image specifications, encapsulates geophysical business software characteristics, and ensures that different container runtimes and tools can interact with the image repository. The specification defines the image format, metadata, and operation methods, making image creation, storage, and distribution a unified standard. Specifically, it includes:

[0062] It provides a RESTful API, allowing developers to interact with the image repository programmatically. For example, operations such as uploading, downloading, and querying images can be implemented through API calls.

[0063] It provides a command-line interface (CLI) tool, allowing users to easily manage images in the terminal. The CLI tool is integrated with the container runtime, enabling users to conveniently use the image repository. The container image repository achieves efficient, secure, and reliable container image storage and distribution through the comprehensive use of storage, distribution, security, and interface and protocol technologies.

[0064] In this invention, a software storage tiering technology was developed for the geophysical image repository. Storage layers are determined based on storage frequency; specifically, frequently accessed geophysical software images are stored in a high-speed storage layer, while less frequently accessed images are stored in a low-cost storage layer, accelerating image retrieval and push speeds. Simultaneously, the repository and images are streamlined to the most efficient level by considering the underlying development and computational dependencies of the geophysical software.

[0065] Furthermore, software isolation in geophysical high-performance computing scenarios has certain industry-specific characteristics. A key challenge is how to ensure different types of computing tasks receive sufficient resources while maximizing resource utilization and avoiding unnecessary resource idleness. This invention combines geophysical scheduling strategies with resource template development and management for jobs with varying resource loads, enabling the rapid generation of computing units for different computing types based on different resource requirement templates. Furthermore, by leveraging namespaces and cgroups, the system calls that geophysical high-performance computing processes within containers can execute are restricted, as are access permissions for files, network resources, etc., preventing unauthorized access to resources on the host system or other containers by computing processes within containers, thus achieving secure environmental isolation between multiple computing units.

[0066] Example 2:

[0067] Step 2: Geophysical Software Mirror Version Control

[0068] Based on actual software business scenarios, specific operations include:

[0069] Step 21: Differentiated construction of different versions of the same geophysical software, that is, creating different mirrors for different software versions and labeling them.

[0070] This invention targets different geophysical software and characteristics, and different types of computing tasks (serial, parallel, IO, TAPE, etc.), and uses technologies such as Dockerfile and Dockercompose to perform targeted tagging construction.

[0071] Step 22: Simplified construction of mirror images from different geophysical software.

[0072] This invention addresses the need for multi-stage, differentiated software image construction for various geophysical high-performance computing scenarios with different construction parameters, conditions, and base images. For example, software requiring large-memory computing tasks necessitates targeted differentiated construction development and image distribution strategy design based on a specific base image, configuration files, and underlying hardware environment.

[0073] Specifically, this invention employs a multi-stage build process, dividing the build process into multiple stages. For example, compilation and building can be performed in one stage, and then in another stage, only the necessary files and dependencies can be copied into the final image, thereby removing unnecessary files and tools generated during the build process.

[0074] Furthermore, in multi-stage builds, smaller intermediate images can be selected for subsequent build steps to reduce the size of the final image.

[0075] In this invention, two other key points regarding the simplified construction of the sphere physics software image are:

[0076] (1) Select a suitable base image

[0077] Choose official, lightweight base images whenever possible to ensure they meet the application's runtime environment requirements and avoid introducing unnecessary dependencies and increased file size due to improper base image selection.

[0078] (2) Reduce the installation of unnecessary software packages

[0079] When building a container image, only the necessary packages for the application to run are installed based on the application's dependencies, avoiding the installation of tools and libraries used during development but unnecessary in the production environment.

[0080] Furthermore, for dependencies that can be dynamically loaded at runtime, use dynamic installation at container startup instead of including them during image building.

[0081] Step 23: Establish a rollback strategy to facilitate version rollback and change tracking.

[0082] Example 3:

[0083] Step 3: Layered Construction of Geophysical Software

[0084] Leveraging the layered nature of imagery can improve build efficiency and save storage space. Specifically, by utilizing the layered nature of container images, shared components (such as the operating system layer) can be placed at the bottom layer, while application-specific layers can be placed at the top layer. This reduces the storage of duplicate data and increases build speed.

[0085] The geophysical mirror image of this invention is mainly divided into the following layers:

[0086] (1) Base Layer: As the lowest layer of the container image, it typically consists of the operating system's basic files and libraries. For example, for a Linux-based container image, the base layer may contain the Linux kernel, basic command-line tools, and system libraries. The base layer provides the basic runtime environment for the container, ensuring that the container can start and run normally.

[0087] (2) Dependency Layers: Above the base layer, a container image may contain multiple dependency layers. These dependency layers contain the packages, libraries, and tools necessary for the application to run. The purpose of dependency layers is to ensure that the application can run correctly in the container and to facilitate the management and updating of the application's dependencies.

[0088] (3) Application Layer: The application layer is the top layer of the container image, containing the actual application code and configuration files. The application layer is typically built by developers and can be customized to meet specific application needs. The application layer can include application binaries, scripts, configuration files, data files, etc. When the container starts, the container runtime loads the application layer content into the container and executes the application.

[0089] (4) Writable Layer: When a container starts, the container runtime creates a writable layer on top of the container image. This writable layer allows the container to modify the file system while running geophysical high-performance computing services, such as writing log files and creating temporary files. The writable layer is temporary; its contents are deleted when the container stops or is deleted. The existence of the writable layer allows the container to make runtime modifications without modifying the container image, thereby improving the container's flexibility and maintainability.

[0090] Furthermore, this invention performs layered construction of geophysical software through underlying OS distribution build scripts, basic environment configuration build scripts, image environment configuration scripts, and multi-scenario distribution configuration scripts.

[0091] Furthermore, by combining a geophysical high-performance computing architecture with a computer multi-level storage structure, this invention achieves a combined storage method of earthquake data buffer storage and PVC technology; in terms of network transmission, it realizes high-speed cloud-based distribution of mirrored data based on the existing TCP / IP high-performance computing network.

[0092] Example 4:

[0093] Step 4: Standardization of Geophysical Software Computing Environment

[0094] like Figure 4As shown, geophysical software is packaged into an image and depends on the user environment, storage environment, software environment, network environment, and operating system environment. To standardize the computing environment for geophysical software, this invention addresses the following aspects:

[0095] (1) Unified environment configuration: Develop a unified environment configuration, define standardized environment and dependencies through Dockerfile, and ensure that the same Docker image is built each time to achieve environment consistency.

[0096] (2) Preset environment variables and configuration files in the image: Make the configuration files part of the image build. Use the COPY instruction in the Dockerfile to copy the configuration files from the host machine to the specified location in the container.

[0097] (3) During runtime, the RUN command is used to execute commands to apply these configurations, ensuring that the software is automatically configured to the correct state during deployment.

[0098] Example 5:

[0099] Step 5: Rapid batch deployment of geophysical software based on mirroring

[0100] like Figure 5 As shown, the image obtained through step four already contains all the necessary dependencies for the application. Geophysical operations employ large-scale, high-performance parallel computing scenarios, and the relevant software needs to be deployed and debugged on a large scale on the computing cluster in a short period of time.

[0101] This invention utilizes Kubernetes container orchestration tools for batch deployment and scaling. By defining Deployments and Services, the lifecycle of containers can be automatically managed, and services can be rapidly scaled as needed.

[0102] Furthermore, based on Kubernetes networking technology, stable, efficient, and large-scale deployment of geophysical HPC based on traditional IB or TCP / IP networks is achieved.

[0103] Furthermore, the present invention involves automated work in the final image release stage, mainly including automated writing of external environment, storage and network configuration, and automated batch configuration of physical resource underlying OS.

[0104] The above technical solution is only one embodiment of the present invention. For those skilled in the art, based on the principles disclosed in the present invention, it is easy to make various types of improvements or modifications, and not limited to the technical solutions described in the specific embodiments of the present invention. Therefore, the foregoing description is only a preferred option and is not restrictive.

Claims

1. A method for creating containerized images of geophysical software, characterized in that: The method includes: Step 1: Package the geophysical software into an image; Step 2: Implement version control for the mirror of the geophysical software; Step 3: Construct layered images from geophysical software; Step four: Standardize the configuration of the geophysical software computing environment; Step 5: Rapid batch deployment of geophysical software mirrors.

2. The method for creating containerized images of geophysical software according to claim 1, characterized in that: The operation in step one includes: packaging the geophysical software and its dependencies into a standalone, portable image; the image is stored in an image repository.

3. The method for creating containerized images of geophysical software according to claim 2, characterized in that: The images are stored in a hierarchical file system structure, and are stored in a high-speed storage layer or a low-cost storage layer according to the access frequency.

4. The method for creating containerized images of geophysical software according to claim 1, characterized in that: Step one also includes: using geophysical scheduling strategies to develop resource templates for jobs with different resource load levels, so that mirrors of different computing types can quickly generate computing units according to different resource requirement templates.

5. The method for creating containerized images of geophysical software according to claim 1, characterized in that: Step one also includes: based on namespace and cgroups features, restricting the system calls that the geophysical high-performance computing process within the container can execute; restricting the access permissions of processes within the container to files and network resources, thereby achieving secure environment isolation between multiple computing units.

6. The method for creating containerized images of geophysical software according to claim 1, characterized in that: Step two includes: Step 21: Different versions of the same geophysical software are used to construct differential tags; Step 22: Simplify and build mirror images from different geophysical software. Step 23: Establish a rollback strategy to facilitate version rollback and change tracking.

7. The method for creating containerized images of geophysical software according to claim 6, characterized in that: The operations in step 21 include: targeted tagging construction based on Dockerfile and Dockercompose technology for different geophysical software and characteristics and different types of computing tasks.

8. The method for creating containerized images of geophysical software according to claim 6, characterized in that: The operations in step 22 include: performing multi-stage image difference construction for different computing scenarios, different construction parameters, different construction conditions, and different base images; and selecting a smaller intermediate image for subsequent construction steps.

9. The method for creating containerized images of geophysical software according to claim 1, characterized in that: The operation in step three includes: using the layered characteristics of the image, dividing the image into a base layer, a dependency layer, an application layer, and a writable layer.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer-executable program, which, when executed by the computer, causes the computer to perform the steps of the geophysical software containerization image creation method as described in any one of claims 1-9.

Images