A seismic batch processing data flow method and framework for multiple programming languages

Abstract

The present invention belongs to the technical field of seismic batch data processing, and discloses a seismic batch data flow method and framework for multiple programming languages. The method is to separate the development of batch processing into a conventional module and an agent module. After the agent module continuously obtains all seismic streams from the conventional module, it forms a seismic data gather in units of data channels, and according to the communication mode required by all seismic data gathers, adopts a corresponding communication protocol, transmits the seismic data gathers to a multi-programming language computing process, and after the multi-programming language computing process completes the calculation, returns the data to the agent module in a corresponding communication mode. The framework consists of an agent module, a process communication protocol, and a multi-programming language computing process. The present invention reduces the programming threshold and development difficulty of developers, allowing developers of different programming languages ​​to devote themselves to the development of GeoEast batch processing, and better serve GeoEast product development and the construction of the "three-in-one" mechanism. The present invention is suitable for developing the application of AI technology in GeoEast.

Description

Technical Field

[0001] The present invention belongs to the batch processing development technology and framework of the control system in the petroleum field, and relates to a method for batch processing of seismic data streams, and specifically to a method and framework for batch processing of seismic data streams oriented to multiple programming languages. Background Art

[0002] The GeoEast system is a large-scale industry application software, mainly used in the field of oil exploration to find oil and natural gas buried underground. It can also be used for the detection of coal and mineral deposits in the coal and geological mining industries. The system has been continuously upgraded and improved since its development in 2003. At present, the product development of the GeoEast system has entered the era of building a "three-sharing" (sharing, competition, and creation) mechanism, as well as the era of AI technology processing data. However, with the widespread application of AI technology in seismic processing and interpretation, the traditional batch processing development model with C / C++ as the programming language can no longer meet the application of AI technology in GeoEast. This is because the mainstream AI framework on the market is mainly based on Python language, and other traditional batch processing development models with C / C++ as the programming language cannot meet the application of AI technology in the GeoEast system. Based on the above reasons, it is urgent to provide a method to enable multiple programming languages ​​to be applicable to the GeoEast system. Summary of the invention

[0003] The purpose of the present invention is to provide a multi-programming language seismic batch processing data stream method to realize the multi-programming language of batch processing modules;

[0004] The second object of the present invention is to provide a data flow framework that can implement a seismic batch processing data flow method in multiple programming languages.

[0005] To achieve the above-mentioned purpose, the technical solution adopted by the multi-programming language seismic batch processing data stream method of the present invention is as follows:

[0006] A seismic batch processing data stream method for multiple programming languages ​​separates the development of batch processing into two processes: a conventional module and an agent module. After the agent module continuously obtains all seismic streams from the conventional module, it forms a seismic data gather based on data channels. Then, according to the communication method required by all seismic data gathers, the seismic data gathers are transmitted to the multi-programming language computing process using the corresponding communication protocol. After the multi-programming language computing process completes the calculation, the data is returned to the agent module in the corresponding communication method, thereby iteratively completing the processing of the entire seismic data.

[0007] As a limitation of the steps of the present invention:

[0008] S1. After the operation process starts to execute, the seismic data stream flows between modules in units of data channels. When the seismic data channel flows from the conventional module to the proxy module, the proxy module repeats the operation of reading the data channel and reads the next data channel from the conventional module until the data channel is the last channel of the channel gather. The proxy module completes the storage of a seismic data channel gather.

[0009] S2. After the seismic data gather is prepared, the proxy module will start the multi-programming language calculation process. After the multi-programming language calculation process is started, it will communicate with the proxy module to realize the data exchange of the seismic data gather. After the multi-programming language calculation process completes the data calculation and generates the result, it will write the data into the proxy module.

[0010] S3, the proxy module will write the result data of the seismic data gather into the next conventional module one by one in units of channels;

[0011] S4, looping through steps S1 to S3 until the seismic data stream is exhausted.

[0012] As a limitation of step S1 of the present invention: step S1 includes the following steps performed in sequence:

[0013] S1.1 Agent module initialization, specifically:

[0014] a1. Start the job, and the agent module runs with the job;

[0015] a2. Initialize the job running environment and call the am function of the proxy module;

[0016] a3. Execute the am function to complete initialization;

[0017] S1.2, the proxy module reads a gather data, including the following steps:

[0018] b1, seismic data channels flow from the conventional module into the proxy module;

[0019] b2. Execute the pm function and cache each track one by one until a track set is cached.

[0020] As a limitation of step S2 of the present invention: step S2 comprises the following steps:

[0021] S2.1, the proxy module starts the multi-programming language computing process, and according to different communication modes, uses the corresponding communication protocol to send the gathers in step b2 to the multi-programming language computing process;

[0022] S2.2, multi-programming language computing process uses different communication modes to read gather data;

[0023] S2.3, multi-programming language computing process initialization, when the multi-programming language computing process receives the first track gather data, it will call the initialization interface in the core interface, and execute the track header word initialization function, construct the read and write header word and allocate the track header word cache space;

[0024] S2.4, multi-programming language computing process data processing;

[0025] S2.5. Multiple programming language computing processes write back gather data.

[0026] As a limitation of step S2.2 of the invention: said step S2.2 includes reading of multi-programming language computing process gathers based on network communication and reading of multi-programming language computing process gathers based on shared memory, wherein reading of multi-programming language computing process gathers based on network communication includes the following steps:

[0027] e1. The multi-programming language computing process parses the command line parameters, obtains the communication port and initializes the communication client object, and constructs the request object according to the communication format. The framework will obtain the corresponding data when the request is initiated for the first time, including the seismic data description and request parameter information;

[0028] e2. Convert the request object into a binary data stream, which is sent by the communication client object to the proxy module;

[0029] e3. After receiving the request, the proxy module converts the data of a track gather into a data stream according to the communication format, and transmits it to the client through the communication object of the server;

[0030] e4. After receiving the response data stream returned by the proxy module, the multi-programming language computing process uses the communication protocol of the multi-programming language computing process to parse the data stream into a response object, thus completing the conversion of a gather data from the C++ object of the proxy module to the Python object of the multi-programming language computing process;

[0031] The read-in of the multi-programming language computing process based on shared memory includes the following steps:

[0032] f1. The proxy module sends a semaphore synchronization notification, and the multi-programming language computing process starts to execute. At the same time, the proxy module enters a waiting state and can only continue to execute after the multi-programming language computing process sends a semaphore synchronization notification;

[0033] f2. Multiple programming language computing processes access shared memory and access shared memory in accordance with the agreed memory layout format;

[0034] f3. The multi-programming language computing process will read the PDL parameters and the header word description table from the shared memory when accessing the shared memory for the first time. In the subsequent shared memory data exchange process, these two parts of data will no longer be read and written;

[0035] f4. The multi-programming language computing process parses the shared memory, reads the sample data and header characters of the seismic gather, and converts the sample data of the seismic gather into Python's NumPy multidimensional array object, and converts the header characters into Python's one-dimensional array, completing the conversion of a gather data from the C++ object of the proxy module to the NumPy object of the multi-programming language computing process.

[0036] As a limitation of step S2.5 of the present invention: step S2.5 includes writing back data of multi-programming language computing processes based on network communication and writing back data of multi-programming language computing processes based on shared memory, wherein writing back data of multi-programming language computing processes based on network communication includes the following steps:

[0037] g1. If the multi-programming language computing process fails to write or the proxy process fails to read, check whether the other process exists. If the process dies unexpectedly, clean up the communication resources and exit the process. Otherwise, continue to execute;

[0038] g2. Convert NumPy to a data byte stream;

[0039] g3. Construct a request format object and write the header word data and track data into the request;

[0040] g4. Use the client's communication object to send the data stream to the proxy module;

[0041] g5. The proxy module receives the write-back data stream and returns an empty response to indicate that the data was written successfully.

[0042] The shared memory-based multi-programming language computing process data write-back comprises the following steps:

[0043] h1. If the proxy process does not receive the semaphore synchronous wake-up of the multi-programming language computing process for a long time, it will detect whether the other process exists. If the process dies unexpectedly, it will clean up the communication resources and exit the process, otherwise it will continue to execute;

[0044] h2. Write the data in NumPy into shared memory;

[0045] h3. If the number of channels changes, write the number of channels to the corresponding position of the shared memory layout;

[0046] h4. The multi-programming language computing process sends a synchronization notification to inform the proxy module that the shared memory data has been written. The proxy module starts to execute downward, and the multi-programming language computing process enters a waiting state, waiting for the proxy module to complete writing the next track set. It will continue to execute after receiving the next track set synchronization semaphore.

[0047] As a limitation of step S3 of the present invention: step S3 comprises the following steps:

[0048] The S3.1 proxy module writes a gather, including the following steps:

[0049] j1. The proxy module modifies the current track data mode to write out a track;

[0050] j2. Iterate and execute the pm function;

[0051] j3. Write each track to the next regular module according to the number of tracks read from the shared memory;

[0052] j4. Until all output channel data are written;

[0053] S3.2, iteratively execute steps S1.2 to S2.5. The iterative process is as follows:

[0054] k1, the proxy module reads a gather from the upper level conventional module;

[0055] k2, the proxy module uses the communication protocol to transmit data to the multi-programming language computing process;

[0056] k3, multi-programming language computing process to process track header and track data;

[0057] k4, the multi-programming language computing process uses the communication protocol to transmit data to the agent module;

[0058] k5, the proxy module writes the result data as a gather into the next regular module;

[0059] S3.3, when the last track gather is transmitted by the proxy module to the multi-programming language computing process, the "last track gather" flag will be written to notify the multi-programming language computing process;

[0060] S3.4, after the multi-programming language computing process completes processing the last track gather, it calls the exit function to release resources, closes the client's communication object, or closes the shared memory object and process synchronization semaphore, and exits the multi-programming language computing process, completing the resource release of the multi-programming language computing process;

[0061] S3.5. After receiving the last processed track set, the proxy process closes the communication object on the server side, or deletes the shared memory object and process synchronization semaphore, and exits the proxy process after the entire job is executed, completing the resource release of the proxy process.

[0062] The technical solution adopted by the multi-programming language seismic batch processing data stream framework of the present invention is as follows:

[0063] A framework for seismic batch data flow for multiple programming languages, including:

[0064] 1. The agent module consists of a basic application library, a communication library and a process management library. The basic application library includes standard error codes, standard trace header word definitions, constant definitions, trace data transmission statistics and common functional functions;

[0065] The communication library implements the basic framework library for communication, which is used to provide communication between processes;

[0066] The process management library implements the management of process startup, process status monitoring and process destruction computing cycle of multi-programming language computing processes;

[0067] 2. Process communication protocol based on network or shared memory

[0068] Implemented a network-based communication framework, a shared memory-based process communication protocol, agent modules, and data exchange between multi-programming language computing processes using the operating system's shared memory;

[0069] 3. Multi-programming language computing process, consisting of a support framework and core interfaces, wherein the support framework includes byte stream data class, header class, data stream parsing class, parameter parsing class and framework processing class;

[0070] Byte stream data flow, oriented to network-based communication methods, used to process byte stream data;

[0071] The path header character class defines standard path header characters, including the size, name and number information of each path header character;

[0072] Data stream parsing class, which is based on network communication and is used to convert data streams into "request" and "response" objects;

[0073] Parameter parsing class, parses command line parameters;

[0074] The framework processing class implements the entire process, including the initialization of the proxy module startup, the organization of the proxy module gather data, the reading, writing and processing of data in multiple programming language computing processes, the writing back of proxy module data, the handling of process exceptions, the encapsulation and parsing of data streams, etc. The framework processing class contains three core interfaces, namely the initialization interface, the processing header character interface and the processing data interface.

[0075] Beneficial Effects

[0076] Due to the adoption of the above technical solution, the present invention has the following technical advances compared with the prior art:

[0077] (1) The framework of the present invention enhances the development function of the existing batch processing module and can support multiple programming languages;

[0078] (2) The framework of the present invention improves the basic capabilities of batch processing in AI, and cutting-edge technologies such as model training and prediction can be widely used in batch processing modules.

[0079] The present invention enables AI technology to be applied to GeoEast processing software in a timely manner, reduces the programming threshold and development difficulty of developers, allows developers of different programming languages ​​to participate in the development of GeoEast batch processing, and better serves GeoEast product development and the construction of the "three-in-one" mechanism. The present invention is suitable for the application of AI technology in GeoEast processing software. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0081] Figure 1 This is a structural organization diagram of Embodiment 1 of the present invention;

[0082] Figure 2 This is a diagram of the data volume format in Example 1 of the present invention;

[0083] Figure 3 This is the format diagram of the first character in Example 1 of the present invention;

[0084] Figure 4 This is a flow chart of Embodiment 2 of the present invention;

[0085] Figure 5 A shared memory object identifier map in Embodiment 2 of the present invention;

[0086] Figure 6 The network communication protocol-request format diagram in Example 2 of the present invention;

[0087] Figure 7 This is a network communication protocol-response format diagram in Example 2 of the present invention;

[0088] Figure 8 This is a shared memory communication-memory layout diagram in Example 2 of the present invention. DETAILED DESCRIPTION

[0089] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0090] Example 1

[0091] This embodiment provides a seismic batch processing data flow framework for multiple programming languages. The framework is introduced by implementing the seismic batch processing data flow framework in the Python programming language. Figure 1 As shown, this embodiment includes:

[0092] 1. The agent module consists of a basic application library, a communication library and a process management library. The basic application library includes standard error codes, standard trace header word definitions, constant definitions, trace data transmission statistics and common functional functions;

[0093] The communication library implements the basic framework library for communication, which is used to provide communication between processes;

[0094] The process management library implements the management of process startup, process status monitoring and process destruction computing cycle of multi-programming language computing processes.

[0095] The above proxy module has the following functions:

[0096] 1. Implemented in traditional C++ batch module development mode, iterating data stream in gather units;

[0097] 2. The "host process" of the multi-programming language computing process is responsible for the life cycle management of the multi-programming language computing process, including the startup and termination of the multi-programming language computing process;

[0098] 3. Serves as a bridge between multi-programming language computing processes and batch module frameworks to realize data flow exchange between multi-programming language computing processes and batch module frameworks.

[0099] 2. Process communication protocol based on network or shared memory

[0100] A network-based communication framework, a shared memory-based process communication protocol, an agent module, and data exchange between multi-programming language computing processes using the operating system's shared memory are implemented.

[0101] The functions of the above process communication protocol are: responsible for data transmission, data format encapsulation and unpacking, etc., specifically: requesting data, returning data, uploading processing results, and returning responses;

[0102] The core data structure of the communication protocol includes the trace data format, trace header data format, seismic data description format, etc. Figure 2 , Figure 3 shown.

[0103] 3. Multi-programming language computing process, consisting of a support framework and core interfaces, wherein the support framework includes byte stream data class, header class, data stream parsing class, parameter parsing class and framework processing class;

[0104] Byte stream data flow, oriented to network-based communication methods, used to process byte stream data;

[0105] The path header character class defines standard path header characters, including the size, name, and number of each path header character;

[0106] Data stream parsing class, which is based on network communication and is used to convert data streams into "request" and "response" objects;

[0107] Parameter parsing class, parses command line parameters;

[0108] The framework processing class implements the entire process, including the initialization of the proxy module startup, the organization of the proxy module gather data, the reading, writing and processing of data in multiple programming language computing processes, the writing back of proxy module data, the handling of process exceptions, the encapsulation and parsing of data streams, etc. The framework processing class contains three core interfaces, namely the initialization interface, the processing header character interface and the processing data interface.

[0109] The main functions of the above multi-programming language computing process are:

[0110] 1. Request data stream or read shared memory from the proxy module;

[0111] 2. Parse data stream or shared memory;

[0112] 3. Process the gather data, for example, using Python's numerical computing library for numerical processing;

[0113] 4. Encapsulate the gather data into a data stream and transmit it back to the proxy module or write it back to the shared memory through the communication protocol.

[0114] Example 2

[0115] This embodiment provides a method for seismic batch processing data streams for multiple programming languages, which is completed using the seismic batch processing data stream framework for multiple programming languages ​​of Example 1. The method is to separate the development of batch processing into two processes: a conventional module and a proxy module. After the proxy module continuously obtains all seismic streams from the conventional module, it forms a seismic data channel set in units of data channels. Then, according to the communication method required by all seismic data channel sets, the seismic data channel set is transmitted to the multi-programming language computing process using the corresponding communication protocol. After the multi-programming language computing process completes the calculation, the data is returned to the proxy module in the corresponding communication method, thereby iteratively completing the processing of the entire seismic data. The specific process is as follows: Figure 4 As shown:

[0116] S1. After the operation process starts, the seismic data stream flows between modules in units of data channels. Figure 1When the conventional module A in the flow into the proxy module, the proxy module will not output the seismic data channel directly to the next conventional module N, but repeat the operation of reading the data channel, reading the next data channel from the conventional module until the data channel is the last channel of the channel set, and the proxy module completes the storage of a seismic data channel set.

[0117] This step specifically includes the following steps performed in sequence:

[0118] S1.1 Agent module initialization, specifically:

[0119] a1. Start the job, and the agent module runs with the job;

[0120] a2. Initialize the job running environment and call the am function of the agent module.

[0121] The proxy module in this embodiment is composed of am function and pm function, wherein the am function is responsible for initialization and the pm function is responsible for iterative operation of data stream.

[0122] a3. Execute the am function to complete initialization; specifically follow the steps below:

[0123] Initialize the PDL parameters of the module;

[0124] Allocate initial gather buffer, this step is to store gather data and trace header data;

[0125] Serialize the PDL parameters into a JSON format string;

[0126] Initialize the execution mode of the multi-programming language computing process. The execution mode is divided into interpreter execution and executable file execution. Interpreter execution will obtain the system's default Python interpreter to execute the Python script file, while executable file execution requires the Python script to be packaged into an independently run executable file using tools and then run.

[0127] S1.2, the proxy module reads a gather data, including the following steps:

[0128] b1, seismic data channels flow from the conventional module into the proxy module;

[0129] b2. Execute the pm function and cache each track one by one until a track set is cached.

[0130] S2, after the seismic data gather is prepared, the proxy module will start the multi-programming language computing process, and after the multi-programming language computing process is started, it will communicate with the proxy module to realize the data exchange of the seismic data gather, and after the multi-programming language computing process completes the data calculation and generates the result, the data will be written into the proxy module. This step S2 includes:

[0131] S2.1, the proxy module starts the multi-programming language computing process, and according to different communication methods, uses the corresponding communication protocol to send the gathers in step b2 to the multi-programming language computing process. This step includes

[0132] 1) Start a multi-programming language computing process based on network communication:

[0133] c1. Determine whether the current multi-programming language computing process has been started. If it has been started, skip this process. Otherwise, create a communication server object and randomly initialize the communication port.

[0134] c2. Perform subsequent operations according to the execution mode of the multi-programming language computing process. If the execution mode is the Python interpreter, obtain the Python interpreter in the current system environment as an executable file, and use the Python execution script and the communication port as parameters to start the process; if the execution mode is an executable file, use the communication port as a command line parameter of the program to directly start the executable file;

[0135] c3. If the multi-programming language computing process fails to start, it will directly exit with an error message. Otherwise, the ID value of the process will be saved as the basis for judging whether the process has been started next time.

[0136] 2) Start a multi-programming language computing process based on shared memory, including:

[0137] d1. Determine whether the current multi-programming language computing process has been started. If it has been started, skip this process;

[0138] d2. Perform subsequent operations according to the execution mode of the multi-programming language computing process. If the execution mode is the Python interpreter, obtain the Python interpreter in the current system environment as an executable file, and use the Python execution script as a parameter to start the process; if the execution mode is an executable file, directly start the executable file;

[0139] d3. If the multi-programming language computing process fails to start, it will directly exit with an error. Otherwise, the ID value of the process will be saved as the basis for judging whether the process has been started next time. Specifically:

[0140] d4. After the multi-programming language computing process is started, it will use the process ID and the proxy process ID as keywords to form a unique identifier to create a shared memory object, as shown in the attached Figure 5 As shown: The name of the semaphore object is organized as sem.GeoEast_ShmBatpDs.<agent process ID>_<child process ID>_Sem;

[0141] d5. The proxy module will wait for the multi-programming language computing process to construct the shared memory object, and then organize the shared memory objects with the same name according to the ID of the multi-programming language computing process to prepare for the subsequent data flow communication;

[0142] d6. The multi-programming language computing process waits for the synchronization semaphore. When the proxy module data is ready, a semaphore synchronization notification is issued and the multi-programming language computing process continues to execute.

[0143] S2.2, the multi-programming language computing process uses the corresponding communication method to read the gather data according to different communication methods; this step includes:

[0144] 1) Multi-programming language computing process gather input based on network communication:

[0145] e1. The multi-programming language computing process parses the command line parameters, obtains the communication port and initializes the communication client object, and constructs the request object according to the communication format. The framework will obtain information such as seismic data description and request parameters when initiating the request for the first time. The communication format requested in this embodiment is as shown in the attached figure. Figure 6 shown.

[0146] e2. Convert the request object into a binary data stream, which is sent by the communication client object to the proxy module;

[0147] e3. After receiving the request, the proxy module converts the data of a track gather into a data stream according to the communication format, and transmits it to the client through the communication object of the server. The communication format of the response in this embodiment is as shown in the attached figure. Figure 7 As shown;

[0148] e4. After receiving the response data stream returned by the proxy module, the multi-programming language computing process uses the communication protocol of the multi-programming language computing process to parse the data stream into a response object, thus completing the conversion of a gather data from the C++ object of the proxy module to the Python object of the multi-programming language computing process;

[0149] 2) Multi-programming language computing process gather input based on shared memory, including the following steps:

[0150] f1. The proxy module sends a semaphore synchronization notification, and the multi-programming language computing process starts to execute. At the same time, the proxy module enters a waiting state and can only continue to execute after the multi-programming language computing process sends a semaphore synchronization notification;

[0151] f2. Multiple programming language computing processes access shared memory and access shared memory in accordance with the agreed memory layout format; the memory layout format of this embodiment accesses shared memory as shown in the attached Figure 8 As shown;

[0152] f3. The multi-programming language computing process will read the PDL parameters and the header word description table from the shared memory when accessing the shared memory for the first time. In the subsequent shared memory data exchange process, these two parts of data will no longer be read and written;

[0153] f4. The multi-programming language computing process parses the shared memory, reads the sample data and header characters of the seismic gather, and converts the sample data of the seismic gather into Python's NumPy multidimensional array object, and converts the header characters into Python's one-dimensional array. This completes the conversion of a gather data from the C++ object of the proxy module to the NumPy object of the multi-programming language computing process.

[0154] S2.3. Initialization of multi-programming language computing process. When the multi-programming language computing process receives the first channel data, it will call the initialization interface in the core interface, execute the header word initialization function, construct the read and write header words, and allocate header word cache space.

[0155] S2.4. Multi-programming language computing process data processing

[0156] After the multi-programming language computing process prepares the sample points and trace header word data, it will execute the trace header word processing and trace data processing interfaces in the core interface. The interface will pass the read trace header word data, write trace header word data and trace data to the Python function in the form of NumPy arrays.

[0157] S2.5, the multi-programming language computing process writes back the gather data. This step includes:

[0158] 1) Data writing back of multi-programming language computing processes based on network communication:

[0159] g1. If the multi-programming language computing process fails to write or the proxy process fails to read, check whether the other process exists. If the process dies unexpectedly, clean up the communication resources and exit the process. Otherwise, continue to execute;

[0160] g2. Convert NumPy to a data byte stream;

[0161] g3. Construct a request format object and write the header word data and track data into the request;

[0162] g4. Use the client's communication object to send the data stream to the proxy module;

[0163] g5. The proxy module receives the write-back data stream and returns an empty response to indicate that the data was written successfully.

[0164] 2) Data writeback of multi-programming language computing processes based on shared memory:

[0165] h1. If the proxy process does not receive the semaphore synchronous wake-up of the multi-programming language computing process for a long time, it will detect whether the other process exists. If the process dies unexpectedly, it will clean up the communication resources and exit the process, otherwise it will continue to execute;

[0166] h2. Write the data in NumPy into shared memory;

[0167] h3. If the number of channels changes, write the number of channels to the corresponding position of the shared memory layout;

[0168] h4. The multi-programming language computing process sends a synchronization notification to inform the proxy module that the shared memory data has been written. The proxy module starts to execute downward, and the multi-programming language computing process enters a waiting state, waiting for the proxy module to complete writing the next track set. It will continue to execute after receiving the next track set synchronization semaphore.

[0169] S3, the proxy module will write the result data of the seismic data gather into the next regular module one by one. This step is as follows:

[0170] The S3.1 proxy module writes a gather, including the following steps:

[0171] j1. The proxy module modifies the current track data mode to write out a track;

[0172] j2. Iterate and execute the pm function;

[0173] j3. Write each channel to the next module according to the number of channels read from the shared memory;

[0174] j4. Until all output channel data are written.

[0175] S3.2, iteratively execute steps 1.2 to 2.5. The iterative process is as follows:

[0176] k1, the proxy module reads a gather from the upper level module;

[0177] k2, the proxy module uses the communication protocol to transmit data to the multi-programming language computing process;

[0178] k3, multi-programming language computing process to process track header and track data;

[0179] k4, the multi-programming language computing process uses the communication protocol to transmit data to the agent module;

[0180] k5. The proxy module writes the result data as a gather to the next regular module.

[0181] S3.3, when the last track gather is transmitted by the proxy module to the multi-programming language computing process, the "last track gather" flag will be written to notify the multi-programming language computing process;

[0182] S3.4, after the multi-programming language computing process completes processing the last track gather, it calls the exit function to release resources, closes the client's communication object, or closes the shared memory object and process synchronization semaphore, and exits the multi-programming language computing process, completing the resource release of the multi-programming language computing process;

[0183] S3.5. After receiving the last processed track set, the proxy process will close the communication object on the server side, or delete the shared memory object and process synchronization semaphore, and exit the proxy process after the entire job is executed, completing the resource release of the proxy process.

[0184] S4. Steps S1 to S3 are executed in a loop until the seismic data stream is exhausted.

Claims

1. A seismic batch processing data stream method for multiple programming languages, characterized in that: The development of batch processing is separated into two processes, the conventional module and the proxy module. After the proxy module continuously obtains all seismic streams from the conventional module, it forms a seismic data gather in units of data channels. Then, according to the communication method required by all seismic data gathers, the seismic data gathers are transmitted to the multi-programming language computing process using the corresponding communication protocol. After the multi-programming language computing process completes the calculation, the data is returned to the proxy module in the corresponding communication method, thereby iteratively completing the processing of the entire seismic data, which includes the following steps performed in sequence: S1. After the operation process starts to execute, the seismic data stream flows between modules in units of data channels. When the seismic data channel flows from the conventional module to the proxy module, the proxy module repeats the operation of reading the data channel and reads the next data channel from the conventional module until the data channel is the last channel of the channel gather. The proxy module completes the storage of a seismic data channel gather. Step S1 includes the following steps performed in sequence: S1.1 Agent module initialization, specifically, a1. Start the job, and the agent module runs with the job; a2. Initialize the job running environment and call the am function of the proxy module; a3. Execute the am function to complete initialization; S1.2, the proxy module reads a gather data, including the following steps: b1, seismic data channels flow from the conventional module into the proxy module; b2. Execute the pm function and cache each trace until a trace set is cached; S2. After the seismic data gather is prepared, the proxy module will start the multi-programming language calculation process. After the multi-programming language calculation process is started, it will communicate with the proxy module to realize the data exchange of the seismic data gather. After the multi-programming language calculation process completes the data calculation and generates the result, it will write the data into the proxy module. Step S2 includes the following steps: S2.1, the proxy module starts the multi-programming language computing process, and according to different communication modes, uses the corresponding communication protocol to send the gathers in step b2 to the multi-programming language computing process; S2.2, multi-programming language computing process uses different communication modes to read gather data; This step includes reading in a multi-programming language computing process gather based on network communication and reading in a multi-programming language computing process gather based on shared memory, wherein reading in a multi-programming language computing process gather based on network communication includes the following steps: e1. The multi-programming language computing process parses the command line parameters, obtains the communication port and initializes the communication client object, and constructs the request object according to the communication format. The framework will obtain the corresponding data when the request is initiated for the first time, including the seismic data description and request parameter information; e2. Convert the request object into a binary data stream, which is sent by the communication client object to the proxy module; e3. After receiving the request, the proxy module converts the data of a track gather into a data stream according to the communication format, and transmits it to the client through the communication object of the server; e4. After receiving the response data stream returned by the proxy module, the multi-programming language computing process uses the communication protocol of the multi-programming language computing process to parse the data stream into a response object, thus completing the conversion of a gather data from the C++ object of the proxy module to the Python object of the multi-programming language computing process; The read-in of the multi-programming language computing process based on shared memory includes the following steps: f1. The proxy module sends a semaphore synchronization notification, and the multi-programming language computing process starts to execute. At the same time, the proxy module enters a waiting state and can only continue to execute after the multi-programming language computing process sends a semaphore synchronization notification; f2. Multiple programming language computing processes access shared memory and access shared memory in accordance with the agreed memory layout format; f3. The multi-programming language computing process will read the PDL parameters and the header word description table from the shared memory when accessing the shared memory for the first time. In the subsequent shared memory data exchange process, these two parts of data will no longer be read and written; f4. The multi-programming language computing process parses the shared memory, reads the sample data and trace header of the seismic gather, and converts the sample data of the seismic gather into a Python NumPy multidimensional array object, and converts the trace header into a Python one-dimensional array, completing the conversion of a gather data from the C++ object of the proxy module to the NumPy object of the multi-programming language computing process; S2.3, multi-programming language computing process initialization, when the multi-programming language computing process receives the first track gather data, it will call the initialization interface in the core interface, and execute the track header word initialization function, construct the read and write header word and allocate the track header word cache space; S2.4, multi-programming language computing process data processing; S2.5, multiple programming language computing processes write back gather data; S3, the proxy module will write the result data of the seismic data gather into the next conventional module one by one in units of channels; S4, looping through steps S1 to S3 until the seismic data stream is exhausted.

2. The multi-programming language oriented seismic batch processing data stream method according to claim 1, characterized in that: Step S2.5 includes writing back the data of the multi-programming language computing process based on network communication and writing back the data of the multi-programming language computing process based on shared memory, wherein writing back the data of the multi-programming language computing process based on network communication includes the following steps: g1. If the multi-programming language computing process fails to write or the proxy process fails to read, check whether the other process exists. If the process dies unexpectedly, clean up the communication resources and exit the process. Otherwise, continue to execute; g2. Convert NumPy to a data byte stream; g3. Construct a request format object and write the header word data and track data into the request; g4. Use the client's communication object to send the data stream to the proxy module; g5. The proxy module receives the write-back data stream and returns an empty response to inform that the data is written successfully. The multi-programming language computing process data write-back based on shared memory includes the following steps: h1. If the proxy process does not receive the semaphore synchronous wake-up of the multi-programming language computing process for a long time, it will detect whether the other process exists. If the process dies unexpectedly, it will clean up the communication resources and exit the process, otherwise it will continue to execute; h2. Write the data in NumPy into shared memory; h3. If the number of channels changes, write the number of channels to the corresponding position of the shared memory layout; h4. The multi-programming language computing process sends a synchronization notification to inform the proxy module that the shared memory data has been written. The proxy module starts to execute downward, and the multi-programming language computing process enters a waiting state, waiting for the proxy module to complete writing the next track set. It will continue to execute after receiving the next track set synchronization semaphore.

3. A multi-programming language oriented seismic batch processing data stream method according to claim 1 or 2, characterized in that: Step S3 includes the following steps: S3.1 The proxy module writes a gather, including the following steps: j1. The proxy module modifies the current track data mode to write out a track; j2. Iterate and execute the pm function; j3. Write each track to the next regular module according to the number of tracks read from the shared memory; j4. Until all output channel data are written; S3.2, iteratively execute steps S1.2 to S2.

5. The iterative process is as follows: k1, the proxy module reads a gather from the upper level conventional module; k2, the proxy module uses the communication protocol to transmit data to the multi-programming language computing process; k3, multi-programming language computing process to process track header and track data; k4, the multi-programming language computing process uses the communication protocol to transmit data to the agent module; k5, the proxy module writes the result data as a gather into the next regular module; S3.3, when the last gather is transmitted from the proxy module to the multi-programming language computing process, the "last gather" flag will be written to notify the multi-programming language computing process; S3.4, after the multi-programming language computing process completes processing the last track gather, it calls the exit function to release resources, closes the client's communication object, or closes the shared memory object and process synchronization semaphore, and exits the multi-programming language computing process, completing the resource release of the multi-programming language computing process; S3.

5. After receiving the last processed track set, the proxy process closes the communication object on the server side, or deletes the shared memory object and process synchronization semaphore, and exits the proxy process after the entire job is executed, completing the resource release of the proxy process.

4. A system for executing the multi-programming language oriented seismic batch processing data stream method as claimed in any one of claims 1 to 3, characterized in that: include:

1. The agent module consists of a basic application library, a communication library and a process management library. The basic application library includes standard error codes, standard trace header word definitions, constant definitions, trace data transmission statistics and common functional functions; The communication library implements the basic framework library for communication, which is used to provide communication between processes; The process management library implements the management of process startup, process status monitoring and process destruction computing cycle of multi-programming language computing processes; 2. Process communication protocol based on network or shared memory Implemented a network-based communication framework, a shared memory-based process communication protocol, an agent module, and data exchange between multi-programming language computing processes using the operating system's shared memory; 3. Multi-programming language computing process, consisting of a support framework and core interfaces, wherein the support framework includes byte stream data class, header class, data stream parsing class, parameter parsing class and framework processing class; Byte stream data flow, oriented to network-based communication methods, used to process byte stream data; The path header character class defines standard path header characters, including the size, name, and number of each path header character; Data stream parsing class, which is based on network communication and is used to convert data streams into "request" and "response" objects; Parameter parsing class, parses command line parameters; The framework processing class implements the entire process, including the startup initialization of the proxy module, the organization of the proxy module gather data, the reading, writing and processing of data in multiple programming language computing processes, the writing back of proxy module data, the handling of process exceptions, and the encapsulation and parsing of data streams. The framework processing class contains three core interfaces, namely the initialization interface, the header character processing interface, and the data processing interface.

Images