Information processing method, program, and information processing device

Abstract

To provide an information processing method, a program, and an information processing device that effectively utilize a computing resource required for pipeline processing.SOLUTION: An acquisition unit 120 acquires a configuration file that defines a pipeline processing order and a parameter that specifies a container that serves as a virtual environment for executing each component. A generation unit 121 generates, on the basis of the parameter, the container in accordance with the pipeline processing order defined in the configuration file. Here, each component defined in the configuration file is defined to calculate a computing resource of the container of an adjacent component adjacent to a downstream of the component in question, and to rewrite the parameter that specifies the computing resource of the container of the adjacent component in the configuration file. The generation unit 121 generates, on the basis of the configuration file rewritten by a container for executing an upstream component in the pipeline processing, a container for executing the adjacent component.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing method, a program, and an information processing device, and more particularly to a technology for managing pipeline processing using container virtualization technology. [Background technology]

[0002] A known method of computational processing using a computer is "pipeline processing," in which processing elements are arranged in series and the output of one element becomes the input of the next element adjacent downstream, thereby performing sequential processing. Patent Document 1 discloses a method of selecting a data processing pipeline for one machine learning (ML) model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-87842 Summary of the Invention [Problem to be solved by the invention]

[0004] Among machine learning methods, in particular transfer learning, to implement it using pipeline processing requires the creation and linking of multiple ML models, which makes the processing flow complicated. For this reason, if transfer learning is implemented using conventional pipeline processing, it is necessary to always reserve computational resources based on the maximum amount of data to be processed, which makes it difficult to use computational resources effectively.

[0005] The present invention has been made in view of these points, and has as its object to provide a technique for effectively utilizing the computational resources required for pipeline processing. [Means for solving the problem]

[0006] A first aspect of the present invention is an information processing method, in which a processor executes the following steps: acquiring a configuration file that defines a processing order of a pipeline process and parameters that specify a container that serves as a virtual environment for executing each component of the pipeline process; generating a container based on the parameters in accordance with the pipeline process order defined in the configuration file; calculating, based on a data size of an output of a component executed by the generated container, computational resources to be allocated to a container of an adjacent component downstream of the component; rewriting, in the configuration file, the parameters that specify the computational resources to be allocated to the container of the adjacent component; and generating a container for executing the adjacent component based on the configuration file with the rewritten parameters.

[0007] Each component defined in the configuration file may be defined to calculate the computational resources to be allocated to the container of an adjacent component adjacent to the downstream side of the component based on the data size of the output of the component, and to rewrite the parameters in the configuration file that specify the computational resources to be allocated to the container of the adjacent component.

[0008] In the information processing method, the processor may further perform the steps of calculating the computational resources to be allocated to the container of the adjacent component adjacent to the downstream side of the component, based on the data size of the output of the component executed by the container generated based on the configuration file with the rewritten parameters, and rewriting the parameters in the configuration file that specify the computational resources to be allocated to the container of the adjacent component.

[0009] The pipeline process may be a process for performing transfer learning, and may include at least (1) a component for generating a representation learning model for target data, (2) a component for generating a representation learning model for source data, (3) a component for generating a representation transfer model, (4) a component for domain adaptation processing, and (5) a component for generating a machine learning model for a task.

[0010] The computing resources may include at least one of a memory to be allocated to the container and the number of processor cores to be allocated to the container.

[0011] A second aspect of the present invention is a program that causes a computer to perform the following functions: acquire a configuration file that defines a processing order of pipeline processing and parameters that specify containers that serve as virtual environments for executing each component of the pipeline processing; generate the containers based on the parameters in accordance with the processing order of pipeline processing defined in the configuration file; calculate, based on the data size of the output of a component executed by the generated container, computational resources to be allocated to a container of an adjacent component downstream of the component; rewrite, in the configuration file, the parameters that specify the computational resources to be allocated to the container of the adjacent component; and generate a container for executing the adjacent component based on the configuration file with the rewritten parameters.

[0012] In order to provide this program or to update a part of the program, a computer-readable recording medium on which this program is recorded may be provided, or this program may be transmitted over a communication line.

[0013] A third aspect of the present invention is an information processing device. The device includes: an acquisition unit that acquires a configuration file that defines a processing order of pipeline processing and parameters that specify containers that serve as virtual environments for executing each component; and a generation unit that generates the containers based on the parameters in accordance with the processing order of the pipeline defined in the configuration file. Here, each component defined in the configuration file is defined to calculate computational resources to be allocated to a container of an adjacent component downstream of the component based on a data size of the component's output, and to rewrite the parameters in the configuration file that specify the computational resources to be allocated to the container of the adjacent component. The generation unit generates a container for executing the adjacent component based on the configuration file rewritten by a container that executes a component upstream in the pipeline processing.

[0014] Any combination of the above components, and any transformation of the present invention into a method, device, system, computer program, data structure, recording medium, etc., are also valid aspects of the present invention. [Effects of the Invention]

[0015] According to the present invention, it is possible to effectively utilize the computational resources required for pipeline processing. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram illustrating container virtualization. [Figure 2] 1 is a schematic diagram for explaining an overview of pipeline processing executed by an information processing device according to an embodiment; [Figure 3] FIG. 10 is a diagram illustrating an example of pipeline processing defined in a configuration file. [Figure 4] FIG. 2 is a diagram schematically illustrating the functional configuration of components executed by each container. [Figure 5] 10 is a flowchart illustrating a flow of information processing executed by an information processing device according to an embodiment. [Figure 6] FIG. 1 is a diagram illustrating pipeline processing applied to transfer learning. DETAILED DESCRIPTION OF THE INVENTION

[0017] <Technology underlying the embodiment> 1 is a diagram for explaining container virtualization. Hereinafter, with reference to FIG. 1, a brief description will be given of the container virtualization technology that is the premise of the processing in the information processing device 1 according to the embodiment.

[0018] The information processing device 1 is hardware having a central processing unit (CPU), a graphics processing unit (GPU), and a storage device including a volatile memory and a non-volatile memory, and is specifically a blade server or a cloud server.

[0019] An information processing device 1 according to an embodiment executes pipeline processing using container virtualization technology. Container virtualization technology is a type of virtualization technology for building a virtual computing environment. Virtualization technology is a technology for abstracting physical computing resources such as servers and dividing and operating the computing resources. Here, "computing resources" include at least one of memory allocated to the virtual computing environment and the number of processor cores allocated to the virtual computing environment. In addition, the computing resources may also include the number of coprocessors, such as GPUs, allocated to the virtual computing environment.

[0020] Conventional virtualization technology, which differs from container virtualization technology, is a technology that starts a guest operating system on a physical server and executes processing on the guest operating system. Conventional virtualization technology builds multiple virtual computing environments on a single physical server by starting multiple guest operating systems on the physical server.

[0021] In contrast, with container virtualization technology, a "container engine" installed on a host operating system running on a physical server creates an independent computing environment called a "container" and runs applications. Container virtualization technology is similar to conventional virtualization technology in that it creates multiple virtual computing environments on a physical server. However, containers, which are virtual computing environments created with container virtualization technology, differ from conventional virtualization technology in that they do not require a guest operating system.

[0022] In the example shown in FIG. 1, the information processing device 1 is a cloud server composed of multiple physical servers. As shown in FIG. 1, a host operating system O runs on the information processing device 1, and a container engine V runs on the host operating system. The container engine V generates one or more containers C based on a configuration file for managing predetermined containers, and runs applications in each container C. FIG. 1 shows that the container engine V manages N containers C (N is a natural number) from a first container C1 to an Nth container Cn.

[0023] Conventional virtualization technology requires a guest OS to be started for each virtual environment, resulting in slow startup times and requiring significant computing resources to operate. In contrast, container virtualization technology requires a host operating system O to directly start containers, eliminating the need for a guest operating system. This eliminates the need for a guest operating system, as well as the startup time and computing resources required to run the guest operating system. In Figure 1, the container engine V and container C form the virtualization layer that realizes the virtual computing environment, but with container virtualization technology, there is no guest operating system in the virtualization layer. This allows container virtualization technology to utilize the computing resources of the information processing device 1 more effectively than conventional virtualization technology.

[0024] <Outline of the embodiment> Based on the above, an outline of an embodiment will be described. An information processing device 1 according to an embodiment treats input data, processing of the input data (executed by an application program), and output of output data as a result of the processing as processing units (hereinafter referred to as "components"), and assigns the components to elements of a pipeline processing to execute calculations. As is well known, pipeline processing is a process in which at least two different components are connected in series, and output data from an upstream component becomes input data for an adjacent downstream component (hereinafter referred to as an "adjacent component"). The information processing device 1 according to an embodiment assigns each component constituting the pipeline processing to a container C and executes each component as an application for the container C.

[0025] Here, the information processing device 1 according to the embodiment is configured so that the component executed by each container C calculates the computational resources to be allocated to adjacent containers based on its own output data. The information processing device 1 according to the embodiment generates containers for the adjacent components based on the calculated computational resources.

[0026] In this way, the information processing device 1 according to the embodiment executes pipeline processing while dynamically changing the computational resources of adjacent containers, because components executed in each container calculate the computational resources of adjacent containers through calculations. This eliminates the need to reserve more computational resources in advance by assuming the maximum amount of data to be processed, and allows computational resources to be adaptively changed and utilized. Furthermore, the information processing device 1 according to the embodiment executes pipeline processing using container virtualization technology, which reduces the overhead caused by guest operating systems. This allows the information processing device 1 according to the embodiment to effectively utilize the computational resources required for pipeline processing.

[0027] <Functional configuration of information processing device 1 according to the embodiment> FIG. 2 is a diagram schematically illustrating the functional configuration of an information processing device 1 according to an embodiment. The information processing device 1 includes a storage unit 10, a communication unit 11, and a control unit 12. In FIG. 2, arrows indicate main data flows, and there may be data flows not shown in FIG. 2. In FIG. 2, each functional block indicates a configuration in functional units, rather than a configuration in hardware (device) units. Therefore, the functional blocks shown in FIG. 2 may be implemented in a single device, or may be implemented separately in multiple devices. Data may be exchanged between functional blocks via any means, such as a data bus, a network, or a portable storage medium.

[0028] The memory unit 10 is a large-capacity storage device such as a ROM (Read Only Memory) that stores the BIOS (Basic Input Output System) of the computer that realizes the information processing device 1, a RAM (Random Access Memory) that serves as the working area of ​​the information processing device 1, an HDD (Hard Disk Drive) or an SSD (Solid State Drive) that stores various information such as application programs such as an operating system and a container engine V, and configuration files that are referenced when the application programs are executed.

[0029] The communication unit 11 is a communication interface for the information processing device 1 to communicate with external devices, and is realized by a known communication module such as a LAN (Local Area Network) module or a Wi-Fi (registered trademark) module. Hereinafter, in this specification, description of the communication unit 11 may be omitted on the assumption that communication between the information processing device 1 and external devices is via the communication unit 11.

[0030] The control unit 12 is a processor such as a CPU or a GPU of the information processing device 1, and functions as an acquisition unit 120 and a generation unit 121 by executing a program stored in the storage unit .

[0031] 2 shows an example in which the information processing device 1 is configured as a single device. However, as described above with reference to FIG. 1, the information processing device 1 may be realized by a plurality of computing resources such as processors and memories, for example, as in a cloud computing system.

[0032] The acquisition unit 120 reads and acquires from the storage unit 10 a configuration file that defines the processing order of the pipeline processing and parameters that define a container that serves as a virtual environment for executing each component that makes up the pipeline processing.

[0033] Fig. 3 is a diagram schematically illustrating an example of pipeline processing defined in a configuration file. Hereinafter, the pipeline processing using the container virtualization technology executed by the information processing device 1 according to the embodiment will be described with reference to Figs. 2 and 3, and Fig. 4, which will be described later.

[0034] Figure 3 illustrates pipeline processing consisting of three pipelines flowing horizontally and one pipeline flowing vertically in Figure 3. Specifically, the pipeline processing shown in Figure 3 includes a pipeline including Process 1-1 and Process 1-2, a pipeline including Process 2-1 and Process 2-2, a pipeline including Process 3, and a pipeline including Process 4 and Process 5. Here, Process 1-1 uses the first input data as input data and the first intermediate data as output data. Also, Process 1-2 uses the first intermediate data as input data and the first intermediate data as output data.

[0035] Furthermore, the 2-1 process uses the second input data as input data and the 2-1 intermediate data as output data. Similarly, the 2-2 process uses the 2-1 intermediate data as input data and the 2-2 intermediate data as output data. The 3rd process uses both the 1-2 intermediate data and the 2-2 intermediate data as input data and the 3rd intermediate data as output data.

[0036] Furthermore, the fourth process uses the third intermediate data as input data and the fourth intermediate data as output data. The fifth process uses the fourth intermediate data as input data and outputs final output data, which is the final output data of this pipeline. In other words, the pipeline process shown in FIG. 3 can also be considered as a process aimed at outputting final output data based on the first input data and the second input data. However, in the pipeline process shown in FIG. 3, multiple intermediate data are generated during the process, and multiple processes are executed to generate each intermediate data. The intermediate data are stored in the storage unit 10 and may be used later.

[0037] Various parameters that define a container C, which is a virtual environment for executing each component that constitutes the pipeline processing, are defined in the configuration file acquired by the acquisition unit 120. These parameters include parameters that define initial setting values ​​of computational resources to be allocated to the container C, programs to be executed in the container C and their execution conditions, and startup conditions of the container C itself (for example, setting values ​​of the container C, initial installation libraries, etc.).

[0038] 3, a component that executes a 1-1 process using the 1st input data and outputting 1-1st intermediate data is assigned to the first container C1. Similarly, a component that executes a 2-1 process using the 2nd input data and outputting 2-1st intermediate data is assigned to the second container C2. The same applies below. For example, a component that executes a 5th process using the 4th intermediate data and outputting final output data is assigned to the 7th container C7.

[0039] The generation unit 121 functions as a container engine V, and specifically generates a container based on parameters in accordance with the pipeline processing order defined in the configuration file acquired by the acquisition unit 120. In the example of pipeline processing shown in FIG. 3, the generation unit 121 first generates a first container C1 using the computational resources of the information processing device 1. As shown in FIG. 3, the 1-1 process and the 2-1 process are independent of each other, and can be executed individually without waiting for each other's processing. Therefore, if the information processing device 1 has sufficient computational resources, the generation unit 121 also generates a second container C2.

[0040] The 1-1 intermediate data, which is the output of the 1-1 process executed by the first container C1, becomes the input of the 1-2 process executed by the third container C3. Therefore, the data size of the 1-1 intermediate data becomes information for determining the computational resources of the information processing device 1 to be allocated to generate the third container C3. For example, if the data size of the 1-1 intermediate data is large, the size of the memory required to store the 1-1 intermediate data becomes large.

[0041] 4 is a diagram schematically illustrating the functional configuration of components executed by each container C. Each container C includes a processing execution unit 100, a calculation unit 101, and a rewriting unit 102. More specifically, the components executed by each container C are defined in a configuration file, and the processing execution unit 100, the calculation unit 101, and the rewriting unit 102 are realized by a computer program executed in a virtual computing environment constructed using the computing resources of the information processing device 1.

[0042] The process execution unit 100 executes the main process of each container C. In the example shown in Fig. 3, the process execution unit 100 in the first container C1 executes a 1-1 process based on the first input data to generate 1-1 intermediate data. Similarly, the process execution unit 100 in the third container C3 executes a third process based on the 1-2 intermediate data and the 2-2 intermediate data to generate third intermediate data.

[0043] The calculation unit 101 calculates the computational resources to be allocated to the containers of adjacent components downstream of the component executed by the container C, based on the data size of the output of the component executed by the container C, i.e., the data size of the output data generated by the processing execution unit 100. For example, the calculation unit 101 in the first container C1 calculates the computational resources to be allocated to the third container C3, which is the container C of the adjacent component, based on the data size of the 1-1 intermediate data.

[0044] The rewriting unit 102 rewrites the parameters that define the computational resources to be allocated to the container of the adjacent component in the setting file that the acquisition unit 120 reads from the storage unit 10 and stores in the main memory of the information processing device 1. When the rewriting unit 102 rewrites the setting file, the processing of the component executed in container C ends.

[0045] When the processing of a certain container C is completed, the generation unit 121 generates a container C for executing an adjacent component based on the configuration file whose parameters have been rewritten by the rewriting unit 102. The generation unit 121 repeats generating containers C until the pipeline processing defined in the configuration file is completed. In this way, the information processing device 1 according to the embodiment executes pipeline processing while dynamically changing the computational resources of adjacent containers in order for each container C to calculate the computational resources of the adjacent containers, and therefore, it is possible to secure computational resources according to the processing results of upstream components and to effectively utilize the computational resources.

[0046] As described above, the configuration file according to the embodiment defines various parameters that define the container C, which serves as a virtual environment for executing each component constituting the pipeline processing. Therefore, the configuration file also defines a calculation unit 101 that calculates the computational resources to be allocated to the container of an adjacent component downstream of a component based on the data size of the component's output, and a rewriting unit 102 that rewrites the parameters in the configuration file that define the computational resources to be allocated to the container of the adjacent component. The configuration file defines the calculation unit 101 and the rewriting unit 102 in addition to the processing execution unit 100, so that the information processing device 1 according to the embodiment can execute pipeline processing while dynamically changing the computational resources of adjacent containers.

[0047] The calculation unit 101 in the adjacent component calculates the computational resources to be allocated to the container of the adjacent component (hereinafter referred to as "adjacent component" for convenience) further downstream of the adjacent component, based on the data size of the output of the component executed by the container C generated by the generation unit 121 based on the parameter-rewritten configuration file. Also, the rewriting unit 102 in the adjacent component rewrites the parameters that define the computational resources to be allocated to the container of the adjacent component in the configuration file.

[0048] In this way, in the pipeline processing performed by the information processing device 1 according to the embodiment, a component rewrites the configuration file for generating an adjacent component, and the adjacent component rewrites the configuration file for generating an adjacent-adjacent component, so that the processing results of each component can be reflected in determining the computational resources to be allocated to the container C for executing the next component.

[0049] <Processing flow of information processing method executed by information processing device 1> 5 is a flowchart for explaining the flow of information processing executed by the information processing device 1 according to the embodiment. The processing in this flowchart starts, for example, when the information processing device 1 is started.

[0050] The acquisition unit 120 acquires a configuration file that defines the processing order of the pipeline processing and parameters that define a container C that serves as a virtual environment for executing each component that makes up the pipeline processing (S2). The generation unit 121 generates a container C based on the parameters of the pipeline processing defined in the configuration file (S4).

[0051] The processing execution unit 100 in container C executes the main task of the component (S6). Until the processing of each component constituting the pipeline processing is completed (No in S8), the calculation unit 101 in container C calculates the computational resources to be allocated to container C of the adjacent component based on the output data of the processing execution unit 100 (S10). The rewriting unit 102 in container C rewrites the setting file so that the parameters in the setting file that define the computational resources to be allocated to container C of the adjacent component become the computational resources calculated by the calculation unit 101 (S12).

[0052] When the rewriter 102 rewrites the configuration file, the process in this flowchart returns to step S4 and continues. When the process of each component constituting the pipeline process is completed (Yes in S8), the process in this flowchart ends.

[0053] <Usage Scenes of Information Processing Device 1 According to the Embodiment> The above has described a method for executing pipeline processing while dynamically allocating computational resources of the container C of the adjacent component. Next, an example of a usage scenario for the pipeline processing executed by the information processing device 1 according to the embodiment will be described.

[0054] The pipeline processing executed by the information processing device 1 according to the embodiment is used, for example, to execute transfer learning. Transfer learning is a machine learning technique for utilizing data from a domain different from the domain to be learned.

[0055] Figure 6 is a diagram showing pipeline processing applied to transfer learning. The example shown in Figure 6 is transfer learning that aims to generate a machine learning model (ML model) by applying knowledge obtained from source data, which is data from a source domain, to target data, which is data from a target domain. Although not limited thereto, for example, the source data is the web browsing history of a group of users whose user attributes are known, and the target data is purchasing information of another group of users (what users with what attributes purchased).

[0056] 6, the first container C1 performs target processing on raw target data, such as data cleansing, deleting unnecessary information, and changing the data format, while the second container C2 performs source processing on source data, such as data cleansing, deleting unnecessary information, and changing the data format.

[0057] The third container C3 is a container C for components that perform target domain representation learning (RL) on target data processed by a target processing process and generate a representation learning model of the target data. As an example of when the target data processed by the target processing process is a matrix, the target domain representation learning is the well-known non-negative matrix factorization (NMF). Similarly, the fourth container C4 is a container C for components that perform source domain representation learning on source data processed by a source processing process and generate a representation learning model of the source data. The source domain representation learning can also be achieved by the well-known NMF.

[0058] The fifth container C5 is a container C for a component that performs representation transformation (RT) processing on the output data of the third container C3 and the fourth container C4 to generate a representation transformation model. As an example of a case where NMF is used for target domain representation learning and source domain representation learning, the representation transformation can be realized by domain application (HeMap) using known matrix decomposition.

[0059] The sixth container C6 is a container C of components for performing domain adaptation (DA) processing on the processing results of the RT processing. DA processing can be realized, for example, by known domain adaptation by example selection (HEGS). The seventh container C7 is a container C of components for performing machine learning for the task and generating a machine learning model for the task. Machine learning for the task can be realized, for example, by known logistic regression.

[0060] In transfer learning, the computational resources required for each container C vary depending on the target data and source data prepared for learning. Each container C rewrites its configuration file to allocate memory equal to or larger than the intermediate data size so that adjacent components can store the data size of the intermediate data it outputs. This allows the information processing device 1 according to the embodiment to dynamically allocate computational resources according to the progress of processing when pipeline processing transfer learning, thereby enabling transfer learning to be performed while efficiently using computational resources.

[0061] <Advantages of the information processing device 1 according to the embodiment> As described above, the information processing device 1 according to the embodiment can effectively utilize the computational resources required for pipeline processing.

[0062] Furthermore, this invention will make it possible to contribute to Goal 9 of the United Nations' Sustainable Development Goals (SDGs), which is "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote innovation and resilience."

[0063] The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by functionally or physically distributing or integrating any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination also have the effects of the original embodiments. [Explanation of symbols]

[0064] 1. Information processing device 10...Storage section 11. Communications Department 12 Control section 120...Acquisition part 121...Generation section

Claims

1. The processor: obtaining a configuration file that defines a processing order of the pipeline processing and parameters that specify a container having a calculation unit and a rewriting unit that serve as a virtual environment for executing each component that constitutes the pipeline processing; generating a container based on the parameters in accordance with a pipeline processing order defined in the configuration file; a step in which the calculation unit calculates, based on a data size of an output of a component executed by the generated container, a calculation resource to be allocated to a container of an adjacent component adjacent to the downstream side of the component; a step in which the rewriting unit rewrites a parameter that defines a computational resource to be allocated to a container of the adjacent component in the setting file with the computational resource calculated by the calculation unit; generating a container for executing the adjacent component based on the configuration file with the rewritten parameters; Information processing methods.

2. Each component defined in the setting file is defined such that the calculation unit calculates the computational resources to be allocated to the container of the adjacent component downstream of the component based on the data size of the output of the component, and the rewriting unit rewrites the parameters in the setting file that specify the computational resources to be allocated to the container of the adjacent component with the computational resources calculated by the calculation unit. The information processing method according to claim 1 .

3. the processor: a step in which the calculation unit calculates, based on a data size of an output of a component executed by the container generated based on the configuration file whose parameters have been rewritten, a calculation resource to be allocated to a container of an adjacent component adjacent to the downstream side of the component; and a step in which the rewriting unit rewrites a parameter in the setting file that defines the computational resources to be allocated to the container of the adjacent component with the computational resources calculated by the calculation unit. The information processing method according to claim 1 .

4. The pipeline process is a process for performing transfer learning, and includes at least (1) a component for generating a representation learning model for target data, (2) a component for generating a representation learning model for source data, (3) a component for generating a representation transfer model, (4) a component for domain adaptation processing, and (5) a component for generating a machine learning model for a task. The information processing method according to any one of claims 1 to 3.

5. The computing resources include at least one of a memory to be allocated to the container and a number of processor cores to be allocated to the container. The information processing method according to any one of claims 1 to 3.

6. On the computer, a function of acquiring a configuration file that defines a processing order of the pipeline processing and parameters that define a container having a calculation unit and a rewriting unit that serve as a virtual environment for executing each component that constitutes the pipeline processing; a function of generating the container based on the parameters in accordance with a pipeline processing order defined in the configuration file; a function of the calculation unit calculating, based on the data size of the output of a component executed by the generated container, the computational resources to be allocated to the container of an adjacent component adjacent to the downstream side of the component; a function of the rewriting unit rewriting parameters that define the computational resources to be allocated to the container of the adjacent component in the setting file to the computational resources calculated by the calculation unit; and generating a container for executing the adjacent component based on the configuration file in which the parameters have been rewritten. program.

7. an acquisition unit that acquires a configuration file that defines a processing order of the pipeline processing and parameters that specify a container having a calculation unit and a rewriting unit that serve as a virtual environment for executing each component; a generation unit that generates the container based on the parameters in accordance with a pipeline processing order defined in the configuration file, Each component defined in the setting file is defined such that the calculation unit calculates the computational resources to be allocated to the container of the adjacent component downstream of the component based on the data size of the output of the component, and the rewriting unit rewrites the parameters in the setting file that define the computational resources to be allocated to the container of the adjacent component with the computational resources calculated by the calculation unit; the generation unit generates a container for executing the adjacent component based on the configuration file rewritten by a container that executes an upstream component in the pipeline processing. Information processing device.

Images