Inference device, inference method, and inference program

Abstract

In an inference device (10), an acquisition unit (15a) acquires, for a plurality of tasks for a prescribed medium, a feature quantity of the medium and a query parameter representing a feature quantity of each task. A prediction generation unit (15d) predicts a task feature quantity of each task for the medium from the query parameters acquired by the acquisition unit (15a), using a first relationship between the query parameters representing the feature quantities of tasks that cross over different tasks and a second relationship between the feature quantity of the medium and the query parameter representing the feature quantity of each task.

Description

Inference device, inference method, and inference program

[0001] The present invention relates to an inference device, an inference method, and an inference program.

[0002] Conventionally, for media such as images, audio, and natural language, inference results for multiple tasks, such as object detection for images, region segmentation, and pose estimation, are sometimes required. As an inference method capable of handling multiple tasks, for example, a method called DETR (Detection TRansformer) is known, which solves object detection using a structure called a Transformer and a parameter that serves as a basis for prediction called an Object Query (see Non-Patent Document 1). The method using a Transformer and an Object Query is known to be effective for various tasks such as pose estimation and region segmentation in addition to object detection.

[0003] X. Zhu, “Deformable DETR: Deformable Transformers for End-to-End Object Detection”, 2021

[0004] However, with conventional techniques, it is difficult to solve multiple tasks simultaneously. For example, although a technique using a Transformer and Object Query is effective for various tasks, it is necessary to create a task-specific model specialized for each task and perform inference. In other words, since a model is created for each task and inference is performed, there is a problem in that costs increase according to the number of tasks.

[0005] The present invention has been made in view of the above, and has as its object to enable efficient execution of inference for a plurality of tasks.

[0006] In order to solve the above-mentioned problems and achieve the object, the inference device of the present invention is characterized by having an acquisition unit that acquires, for a plurality of tasks for a specified medium, features of the medium and parameters representing the features of each task, and a prediction unit that predicts task features of each task in the medium from the parameters acquired by the acquisition unit, using a first relationship between the parameters representing the features of each task across different tasks, and a second relationship between the features of the media and the parameters representing the features of each task.

[0007] According to the present invention, it is possible to efficiently execute inference for a plurality of tasks.

[0008] FIG. 1 is a diagram for explaining an overview of the inference device of this embodiment. FIG. 2 is a diagram for explaining an overview of the inference device of this embodiment. FIG. 3 is a diagram for explaining an overview of the inference device of this embodiment. FIG. 4 is a schematic diagram illustrating a general configuration of the inference device of this embodiment. FIG. 5 is a diagram for explaining processing by the inference device. FIG. 6 is a diagram for explaining processing by the inference device. FIG. 7 is a flowchart showing an inference processing procedure. FIG. 8 is a flowchart showing an inference processing procedure. FIG. 9 is a flowchart showing an inference processing procedure. FIG. 10 is a flowchart showing an inference processing procedure. FIG. 11 is a flowchart showing an inference processing procedure. FIG. 12 is a diagram for explaining a second embodiment. FIG. 13 is a flowchart showing an inference processing procedure of the second embodiment. FIG. 14 is a diagram for explaining an example. FIG. 15 is a diagram showing an example of a computer that executes an inference program.

[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are designated by the same reference numerals.

[0010] [Overview of the Inference Apparatus] FIG. 1 is a diagram illustrating an overview of the inference apparatus of this embodiment. In this embodiment, media refers to, for example, images, audio, or natural language. FIG. 1 illustrates a method that uses a Transformer and an Object Query to predict a single task for media. In this method, a model for, for example, an image consists of a convolutional neural network (CNN) part called a backbone created for image classification, and a part called a task head that predicts each task. Then, the image features that are the output of the backbone and the Object Query that are parameters related to the task are input to the Transformer to obtain the output of each task.

[0011] Here, the Transformer is composed of a Transformer Encoder and a Transformer Decoder, as exemplified in Fig. 1. The Transformer Encoder is designed to be common to each model of multiple tasks, and refines backbone image features through nonlinear transformation using Attention, which is a mechanism for learning noteworthy parts of the input by extracting features based on the similarity between input features.

[0012] The Transformer Decoder also receives the image features output from the Transformer Encoder and the Object Query as input, and repeats processing for multiple layers (L layers). As a result, each Object Query gradually predicts the desired task while grasping its own prediction state and recognizing its role in the image.

[0013] This Transformer Decoder is mainly composed of Self-Attention and Cross-Attention. Self-Attention extracts the relationship between predictions corresponding to each of a plurality of Object Queries. Cross-Attention extracts the relationship between image features and predictions corresponding to each of a plurality of Object Queries. Here, Attention is a mechanism that performs nonlinear transformation by extracting features based on the similarity between input features.

[0014] Self-Attention processing is often almost the same for each task model, while Cross-Attention processing often differs depending on the task and method. Therefore, while a method using a Transformer is generally effective for a variety of tasks, it is necessary to create a model specialized for one task.

[0015] Therefore, the inference device of this embodiment provides a specific improvement over the conventional method using Transformer and Object Query, which required creating task-specific models for each task to perform inference, resulting in costs corresponding to the number of tasks, and represents an advancement in the technical field related to multitask AI.

[0016] 2 and 3 illustrate processing in the inference device of this embodiment. Specifically, as illustrated in Fig. 2, in the inference device, the Transformer Decoder and Object Query are extended to multitask in order to solve multiple tasks simultaneously.

[0017] That is, as shown in Fig. 3(b), the inference device prepares the same number of types of Object Queries as the number of tasks, and inputs them all together into the Self-Attention of the Transformer Decoder, and performs processing to extract the relationships between predictions of the Object Queries. This makes it possible to extract the relationships between predictions of multiple tasks.

[0018] Furthermore, in Cross-Attention (CA), the process branches as many times as the number of tasks, and for each type of Object Query corresponding to each task, a process is performed to extract the relationship between image features and the prediction of the Object Query. This makes it possible to incorporate Cross-Attention techniques that have been developed independently for each task. In this way, the inference device uses a multi-task Object Query to perform Self-Attention processing across tasks in the Transformer Decoder and Cross-Attention processing unique to the branched task. This makes it possible to perform multi-task inference more efficiently than the conventional processing for a single task illustrated in FIG. 3(a).

[0019] [Configuration of the inference device] Fig. 4 is a schematic diagram illustrating the general configuration of the inference device of this embodiment. Fig. 5 and Fig. 6 are diagrams for explaining the processing of the inference device. First, as illustrated in Fig. 4, the inference device 10 of this embodiment is realized by a general-purpose computer such as a personal computer, and includes an input unit 11, an output unit 12, a communication control unit 13, a storage unit 14, and a control unit 15.

[0020] The input unit 11 is realized using input devices such as a keyboard and a mouse, and inputs various instruction information such as a command to start processing to the control unit 15 in response to input operations by an operator. The output unit 12 is realized by a display device such as a liquid crystal display, a printing device such as a printer, etc. For example, the output unit 12 displays the results of the inference processing described below.

[0021] The communication control unit 13 is realized by a NIC (Network Interface Card) or the like, and controls communication between the control unit 15 and external devices via telecommunication lines such as a LAN (Local Area Network) or the Internet. For example, the communication control unit 13 controls communication between the control unit 15 and a management device or the like that manages various types of information.

[0022] The storage unit 14 is realized by a semiconductor memory element such as RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or optical disk. The storage unit 14 stores in advance the processing program that operates the inference device 10, data used during execution of the processing program, and the like, or temporarily stores the data each time processing is performed. The storage unit 14 may be configured to communicate with the control unit 15 via the communication control unit 13.

[0023] In this embodiment, the storage unit 14 stores an image 14 a, an image feature generation parameter 14 b, an image feature refinement parameter 14 c, a prediction generation parameter 14 d, a query parameter 14 e, and the like, which are used in the inference process described below. This information is acquired from a management device or the like that manages various types of information via the input unit 11 or the communication control unit 13 before the inference process described below or during the inference process, and is stored in the storage unit 14.

[0024] The image 14a stores an image to be processed and input to the inference process described later.

[0025] The image feature generation parameters 14b store image feature generation parameters used in the inference process described below. The image feature generation parameters are parameters of a neural network model and are obtained by supervised learning of a general neural network using task learning data.

[0026] The image feature refinement parameters 14c store image feature refinement parameters used in the inference process described below. The image feature refinement parameters are parameters of a neural network model, and like the image feature generation parameters, they are obtained by supervised learning of a general neural network using task learning data.

[0027] The prediction generation parameters 14d store prediction generation parameters used in the inference process described below. The prediction generation parameters are parameters of a neural network model, and like the image feature generation parameters, they are obtained by supervised learning of a general neural network using task learning data.

[0028] The query parameters 14e store query parameters used in the inference process described below. The query parameters are parameters that represent the feature quantities of a task, such as an Object Query, and the same number of query parameters as the number N of tasks are stored corresponding to each of the multiple tasks. In the example shown in Fig. 5, N types of query parameters are stored, including a task 1 query parameter, a task 2 query parameter, ..., a task N query parameter.

[0029] The query parameters for each task are randomly initialized and obtained through typical supervised learning of neural networks. The query parameters are vectors with the same dimensions as the vectors processed in the inference process described below.

[0030] The control unit 15 is realized using a CPU (Central Processing Unit), NP (Network Processor), FPGA (Field Programmable Gate Array), etc., and executes a processing program stored in memory. As a result, the control unit 15 functions as an acquisition unit 15a, an image feature generation unit 15b, an image feature refinement unit 15c, a prediction generation unit 15d, an inter-prediction relationship extraction unit 15e, and an image-prediction relationship extraction unit 15f, as exemplified in FIG. 4, to perform inference processing. Note that these functional units may each be implemented in different hardware, or some of them may be implemented in different hardware. The control unit 15 may also include other functional units.

[0031] The acquisition unit 15a acquires, via the input unit 11 or the communication control unit 13, the image to be processed, parameters for generating image features, parameters for refining image features, parameters for generating predictions, parameters for queries, etc., which are used in the inference processing described below, from the memory unit 14.

[0032] Instead of the acquiring unit 15a, the functional units described below may acquire the above information from the storage unit 14 when performing processing.

[0033] 5, the image feature generating unit 15b uses the image feature generating parameters acquired from the storage unit 14 and the image to be processed to generate image feature amounts for the image to be processed based on the image feature generating parameters. The image feature generating unit 15b corresponds to, for example, a Transformer Encoder.

[0034] Here, the image to be processed is acquired as a three-dimensional vector of height x width x RGB. When processing multiple images, the image feature generating unit 15b only needs to acquire the image feature generation parameters once, at the beginning.

[0035] 5, the image feature refinement unit 15c further refines the image feature quantities generated by the image feature generation unit 15b based on image feature refinement parameters acquired from the storage unit 14. Specifically, the image feature refinement unit 15c corresponds to a Transformer Encoder, and refines the feature quantities of the media by performing nonlinear transformation based on Attention, which is a mechanism for further extracting feature quantities using the similarity between the feature quantities of the media.

[0036] Here, when the image feature refinement unit 15c processes the image feature amounts of a plurality of images, it is sufficient to obtain the image feature refinement parameters only once, at the beginning.

[0037] The acquiring unit 15a may acquire refined image features and transfer them to the prediction generating unit 15d, which will be described below. In this case, the acquiring unit 15a acquires media features and task parameters representing the features of each task for a plurality of tasks for a predetermined media, such as an image.

[0038] The prediction generation unit 15d predicts the task features of each task in the media from the query parameters acquired by the acquisition unit 15a, using a first relationship between the query parameters representing the features of each task across different tasks, and a second relationship between the features of the media and the query parameters representing the features of each task.

[0039] Specifically, as illustrated in FIG. 5, the prediction generation unit 15d predicts a desired task using the prediction generation parameters and query parameters acquired from the memory unit 14, and the image features refined by the image feature refinement unit 15c.

[0040] Here, when the prediction generation unit 15d performs processing on image feature amounts of a plurality of images, it is sufficient to obtain the prediction generation parameters and the query parameters only once, at the beginning.

[0041] 5, there are query parameters corresponding to a plurality of tasks. For example, when two tasks, object detection and pose estimation, are targeted for inference processing, query parameters for object detection and query parameters for pose estimation are input to prediction generation unit 15d, and prediction results for the two tasks, object detection and pose estimation, are output from the image feature amounts refined by image feature refinement unit 15c.

[0042] The prediction generation unit 15d predicts task features by repeatedly converting parameters using the similarity between query parameters as a first relationship, and converting parameters using the similarity between media features and query parameters as a second relationship.

[0043] That is, the prediction generation unit 15d performs nonlinear transformation of the query parameters of each task based on an attention mechanism that further extracts feature amounts using the similarity between the query parameters of each task, and performs nonlinear transformation of the query parameters of each task based on an attention mechanism that further extracts feature amounts using the similarity between the feature amounts of the media and the query parameters of each task. The prediction generation unit 15d repeats this process to predict the task feature amounts of each task in the media.

[0044] Specifically, the prediction generation unit 15d is a decoder of a transformer that is a pre-trained neural network, and extracts the first relationship by a self-attention mechanism and the second relationship by a cross-attention mechanism.

[0045] That is, the prediction generation unit 15d is a decoder of a transformer, and Self-Attention extracts the relationship between the query parameters of each task across different tasks, and Cross-Attention extracts the relationship between the media features and the query parameters of each task for each task, thereby predicting the task features of each task in the media.

[0046] Here, Figure 6 is a diagram for explaining the processing of the prediction generation unit. As illustrated in Figure 6, the prediction generation unit 15d includes an inter-prediction relationship extraction unit 15e and an image-inter-prediction relationship extraction unit 15f. The inter-prediction relationship extraction unit 15e corresponds to, for example, Self-Attention of a Transformer Decoder, and performs the same processing across tasks regardless of the type or number of tasks. On the other hand, the image-inter-prediction relationship extraction unit 15f corresponds to, for example, Cross-Attention of a Transformer Decoder, and performs different processing depending on the type of task, so that the same number of image-inter-prediction relationship extraction units 15f as the number of tasks perform processing in parallel.

[0047] The prediction generation unit 15d has a predetermined number L of layers, each layer consisting of a pair of an inter-prediction relationship extraction unit 15e and an image-inter-prediction relationship extraction unit 15f, and the query parameters gradually change as the unit passes through each layer, until a prediction is output as the inference result for each task.

[0048] Specifically, first, in the first layer, the inter-prediction relationship extraction unit 15e extracts relationships between query parameters based on the prediction generation parameters, and updates each query parameter based on the extracted relationships (hereinafter, the updated query parameters may be referred to as "query parameters (predictions)"). Next, the image-prediction relationship extraction unit 15f extracts relationships between image features and the updated query parameters (predictions) of each task based on the prediction generation parameters, and updates the query parameters of each task based on the extracted relationships.

[0049] Next, in the second layer, the inter-prediction relationship extraction unit 15e extracts relationships between each query parameter (prediction) based on the updated prediction generation parameters, and updates each query parameter based on the extracted relationships. Next, the image-prediction relationship extraction unit 15f extracts relationships between image features and the updated query parameters (prediction) of each task based on the prediction generation parameters, and updates the query parameters of each task based on the extracted relationships.

[0050] In this way, the inter-prediction relationship extraction unit 15e and the image-prediction relationship extraction unit 15f repeat the same process up to the Lth layer, thereby outputting a prediction that is the inference result for each task. In this way, the inference device 10 can efficiently execute inference for multiple tasks for media. Furthermore, learning of various parameters can also be executed efficiently because it can be performed for multiple tasks collectively.

[0051] [Inference Processing] Next, the inference processing by the inference device 10 according to this embodiment will be described with reference to Figures 7 to 11. Figures 7 to 11 are flowcharts showing the procedure of the inference processing. First, in Figure 7, the flowchart starts, for example, when the user performs an operation input to instruct the start of the processing.

[0052] First, the acquisition unit 15a acquires from the storage unit 14 the image to be processed, image feature generation parameters, image feature refinement parameters, prediction generation parameters, query parameters, and the like, which are used in the inference process.

[0053] Then, the image feature generating unit 15b uses the image feature generating parameters and the image to be processed to generate image feature amounts for the image to be processed based on the image feature generating parameters (step S1).

[0054] The image feature refinement unit 15c further refines the image feature quantity generated by the image feature generation unit 15b based on the image feature refinement parameters (step S2).

[0055] Next, the prediction generating unit 15d generates a prediction of the feature of each task in the image using the refined image feature and the query parameters for each task, using the prediction generating parameters (step S3), and outputs the generated prediction (step S4). This completes the series of inference processes.

[0056] Next, Fig. 8 illustrates the processing procedure of the image feature generation unit in step S1 shown in Fig. 7. First, the image feature generation unit 15b acquires image feature generation parameters and an input image to be processed from the acquisition unit 15a or the storage unit 14 (steps S101 to S102).

[0057] Next, the image feature generating unit 15b generates image features for the input image to be processed based on the image feature generating parameters (step S103), and outputs the generated image features to the downstream image feature refining unit 15c (step S104), thereby completing the series of processes.

[0058] Next, Fig. 9 illustrates the processing procedure of the image feature generating unit in step S2 shown in Fig. 7. First, the image feature refining unit 15c acquires parameters for image feature refinement from the acquiring unit 15a or the storage unit 14 (step S201), and acquires the image feature amount generated by the image feature generating unit 15b (step S202).

[0059] Next, the image feature refinement unit 15c refines the image feature based on the image feature refinement parameters (step S203). The image feature generation unit 15b outputs the refined image feature to the subsequent prediction generation unit 15d (step S204), thereby completing the series of processes.

[0060] Next, Fig. 10 illustrates the processing procedure of the prediction generation unit in steps S3 and S4 shown in Fig. 7. First, the prediction generation unit 15d acquires prediction generation parameters and query parameters from the acquisition unit 15a or the storage unit 14 (steps S301 and S302), and acquires refined image features from the image feature refinement unit 15c (step S303).

[0061] Next, the prediction generation unit 15d generates a prediction of the feature of the task based on the prediction generation parameters for the image feature and the query parameters (step S304).The prediction generation unit 15d then outputs the generated task prediction (step S305), and ends the series of processes.

[0062] 11 illustrates a detailed processing procedure of the processing of step S304 shown in FIG. 10. After acquiring refined image features from the image feature refinement unit 15c in the processing of step S303, the prediction generation unit 15d performs the first-layer processing shown in FIG. 6. Specifically, the inter-prediction relationship extraction unit 15e extracts relationships between query parameters for each task based on the prediction generation parameters, and updates each query parameter based on the extracted relationships (step S401). Next, the image-prediction relationship extraction unit 15f extracts relationships between the image features and the updated query parameters for each task, and further updates the query parameters for each task based on the extracted relationships (step S402).

[0063] Next, the prediction generation unit 15d performs second-layer processing. That is, the inter-prediction relationship extraction unit 15e extracts relationships between query parameters for the updated query parameters of each task based on the prediction generation parameters, and updates each query parameter based on the extracted relationships. Also, the image-prediction relationship extraction unit 15f extracts relationships between image features and the updated query parameters of each task, and further updates the query parameters of each task based on the extracted relationships.

[0064] In this way, the prediction generation unit 15d repeats the same process up to the Lth layer (step S403), and sets the query parameters updated in the Lth layer as the prediction of the generated task (step S404). Thereafter, the prediction generation unit 15d outputs the prediction of the generated task (step S305), and ends the series of processes.

[0065] It should be noted that the media in the above processing is not limited to images. As mentioned above, similar processing can also be applied to speech and natural language as media. In that case, general feature extraction can be performed when processing each media, and the extracted features can be used as input to the model.

[0066] 12 is a diagram for explaining the processing of an inference device according to a second embodiment. Note that, below, only the differences from the inference processing of the inference device 10 according to the first embodiment will be explained, and explanations of commonalities will be omitted.

[0067] Like the inference device 10 of the above embodiment, the inference device 10 of the second embodiment also provides a specific improvement over the conventional method using a Transformer and Object Query, which required creating task-specific models for each task to perform inference, resulting in costs corresponding to the number of tasks, and represents an advancement in the technical field related to multitask AI.

[0068] Specifically, in the inference device 10 of the above embodiment, as shown in Fig. 6, in the prediction generation unit 15d, the inter-prediction relationship extraction unit 15e performs processing and then the image-prediction relationship extraction unit 15f performs processing, but this processing order is not limited to this. That is, as shown in Fig. 12, the image-prediction relationship extraction unit 15f may perform processing first, and then the inter-prediction relationship extraction unit 15e may perform processing.

[0069] In this case, as with the inference device 10 of the above embodiment, it is possible to efficiently execute inference of multiple tasks for media. Furthermore, since learning of various parameters can be performed for multiple tasks collectively, it is also possible to execute it efficiently.

[0070] Fig. 13 is a flowchart showing the inference processing procedure of the second embodiment. As shown in Fig. 13, the processing of steps S411 to S413 differs from the flowchart of Fig. 11.

[0071] Specifically, after acquiring refined image features from the image feature refinement unit 15c in the process of step S303, the prediction generation unit 15d performs the first-layer process shown in Fig. 12. Specifically, the image-prediction relationship extraction unit 15f extracts relationships between image features and query parameters for each task, and updates each query parameter based on the extracted relationships (step S411). Next, the prediction relationship extraction unit 15e extracts relationships between query parameters for the updated query parameters for each task based on the prediction generation parameters, and updates each query parameter based on the extracted relationships (step S412).

[0072] Next, the prediction generation unit 15d performs second-layer processing. That is, the image-prediction relationship extraction unit 15f extracts relationships between image features and the updated query parameters of each task, and updates the query parameters of each task based on the extracted relationships. Furthermore, the prediction relationship extraction unit 15e extracts relationships between query parameters for the updated query parameters of each task based on the prediction generation parameters, and updates each query parameter based on the extracted relationships.

[0073] In this way, the prediction generation unit 15d repeats the same process up to the Lth layer (step S413), and sets the query parameters updated in the Lth layer as the prediction of the generated task (step S404). Thereafter, the prediction generation unit 15d outputs the prediction of the generated task (step S305), and ends the series of processes.

[0074] [Effects] As described above, in the inference device 10 of this embodiment, the acquisition unit 15a acquires, for a plurality of tasks for a predetermined medium, feature quantities of the media and query parameters representing the feature quantities of each task. The prediction generation unit 15d predicts task feature quantities of each task in the media from the query parameters acquired by the acquisition unit 15a, using a first relationship between the query parameters representing the feature quantities of each task across different tasks, and a second relationship between the feature quantities of the media and the query parameters representing the feature quantities of each task.

[0075] Specifically, the prediction generation unit 15d predicts task features by repeatedly converting parameters using the similarity between query parameters as the first relationship, and converting parameters using the similarity between media features and query parameters as the second relationship.

[0076] For example, the prediction generation unit 15d is a decoder of a transformer that is a pre-trained neural network, and extracts the first relationship by a self-attention mechanism and the second relationship by a cross-attention mechanism.

[0077] This allows for efficient inference of multiple tasks. Furthermore, learning of various parameters for multiple tasks can be performed simultaneously, making it possible to efficiently execute the tasks.

[0078] The media may be images, sounds, or natural languages. In this way, the inference device 10 can efficiently perform inference processing for multiple tasks in a variety of media.

[0079] [Example] Fig. 14 is a diagram for explaining an example. In this example, the tasks were object detection and pose estimation, and the inference time and learning time were compared between a model using a conventional Transformer and the inference device 10 of the above embodiment. As a model using a conventional Transformer, Deformable DETR was used for object detection, and PETR (Position Embedding TRansformation) was used for pose estimation. R-50 was also used as the backbone. In contrast, a comparison was made between a case where R-50 was used as the backbone of the model of the above embodiment, and a case where Swin-T was used.

[0080] 14, when the backbone is the same R-50, it was confirmed that the inference device 10 of the above embodiment can reduce both the training time and inference time to 70% of the conventional time, while achieving an accuracy of 90% or more. Furthermore, when a backbone Swin-T, which is larger than R-50, is used, it was confirmed that the inference device 10 can achieve excellent accuracy with training time and inference time that are 80% of the conventional time.

[0081] [Program] A program describing the processing executed by the inference device 10 according to the above embodiment in a computer-executable language can also be created. In one embodiment, the inference device 10 can be implemented by installing an inference program that executes the above inference processing as package software or online software on a desired computer. For example, by executing the above inference program on an information processing device, the information processing device can function as the inference device 10. The information processing device referred to here includes desktop and notebook personal computers. Other examples of information processing devices include mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as slate terminals such as PDAs (Personal Digital Assistants). The functions of the inference device 10 may also be implemented on a cloud server.

[0082] 15 is a diagram showing an example of a computer that executes an inference program. The computer 1000 includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0083] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1031. The disk drive interface 1040 is connected to a disk drive 1041. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1041. The serial port interface 1050 is connected to a mouse 1051 and a keyboard 1052, for example. The video adapter 1060 is connected to a display 1061, for example.

[0084] Here, the hard disk drive 1031 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. The various pieces of information described in the above embodiments are stored in the hard disk drive 1031 or the memory 1010, for example.

[0085] The inference program is stored in the hard disk drive 1031, for example, as a program module 1093 in which instructions to be executed by the computer 1000 are written. Specifically, the program module 1093 in which each process executed by the inference device 10 described in the above embodiment is written is stored in the hard disk drive 1031.

[0086] Furthermore, data used for information processing by the inference program is stored as program data 1094, for example, in the hard disk drive 1031. Then, the CPU 1020 reads the program module 1093 and the program data 1094 stored in the hard disk drive 1031 into the RAM 1012 as necessary, and executes each of the above-described procedures.

[0087] The program module 1093 and program data 1094 related to the inference program are not limited to being stored in the hard disk drive 1031, but may be stored in a removable storage medium, for example, and read by the CPU 1020 via the disk drive 1041. Alternatively, the program module 1093 and program data 1094 related to the inference program may be stored in another computer connected via a network such as a LAN or a WAN (Wide Area Network), and read by the CPU 1020 via the network interface 1070.

[0088] The following additional notes are provided regarding the above-described embodiments.

[0089] (Supplementary Item 1) An inference device including: a memory; and at least one processor connected to the memory, wherein the processor acquires, for a plurality of tasks for a predetermined medium, feature quantities of the medium and parameters representing feature quantities of each task, and predicts task feature quantities of each task for the medium from the acquired parameters using a first relationship between the parameters representing feature quantities of each task across different tasks and a second relationship between the feature quantities of the media and the parameters representing feature quantities of each task.

[0090] (Supplementary Item 2) A non-transitory storage medium storing a program executable by a computer to perform the inference processing by the inference device according to Supplementary Item 1.

[0091] Although the present invention has been described above as an embodiment, the present invention is not limited to the description and drawings that form part of the disclosure of the present invention. In other words, other embodiments, examples, and operational techniques that can be made by those skilled in the art based on the present invention are all included in the scope of the present invention.

[0092] 10 Inference device 11 Input unit 12 Output unit 13 Communication control unit 14 Storage unit 14a Image 14b Image feature generation parameters 14c Image feature refinement parameters 14d Prediction generation parameters 14e Query parameters 15 Control unit 15a Acquisition unit 15b Image feature generation unit 15c Image feature refinement unit 15d Prediction generation unit (prediction unit) 15e Inter-prediction relationship extraction unit 15f Image-prediction relationship extraction unit

Claims

1. An inference device comprising: an acquisition unit that acquires, for a plurality of tasks for a specified medium, features of the medium and parameters representing the features of each task; and a prediction unit that predicts task features of each task in the medium from the parameters acquired by the acquisition unit, using a first relationship between the parameters representing the features of each task across different tasks, and a second relationship between the features of the media and the parameters representing the features of each task.

2. The inference device described in claim 1, characterized in that the prediction unit predicts the task features by repeatedly transforming the parameters using the similarity between the parameters as the first relationship, and transforming the parameters using the similarity between the media features and the parameters as the second relationship.

3. The inference device described in claim 1, characterized in that the prediction unit is a decoder of a Transformer which is a pre-trained neural network, and the first relationship is extracted by a Self-Attention mechanism and the second relationship is extracted by a Cross-Attention mechanism.

4. The inference device according to claim 1, wherein the media is any one of an image, a voice, and a natural language.

5. An inference method executed by an inference device, comprising: an acquisition step of acquiring, for a plurality of tasks for a specified medium, features of the medium and parameters representing the features of each task; and a prediction step of predicting task features of each task in the medium from the parameters acquired in the acquisition step, using a first relationship between the parameters representing the features of each task across different tasks, and a second relationship between the features of the media and the parameters representing the features of each task.

6. An inference program for causing a computer to function as the inference device according to any one of claims 1 to 4.

Images