Inference device and inference method

Abstract

This helps to suppress the reduction in the effectiveness of computational complexity reduction, while also suppressing the decrease in inference accuracy. [Solution] The inference means includes an input size identification means that identifies the size of each of a plurality of input data; a reference size determination means that determines a reference size; an input data transformation means that transforms the input data based on the reference size to generate a plurality of transformed data; a data joining means that combines the transformed data into a single data; and an inference means that performs inference using the single data as input.

Description

[Technical Field]

[0001] This disclosure relates to an inference device and inference method related to multimodal machine learning. [Background technology]

[0002] One inference technique for neural networks is multimodal processing, which handles multiple types of input data simultaneously. By using multimodal processing, inference accuracy can be improved by integrating and processing multiple input data sources.

[0003] Two representative methods for integrating input data are Early fusion and Late fusion. Early fusion is a method that combines multiple input data before inference is performed by the neural network. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Chi Thang Duong, et al., "Multimodal Classification for Analysing Social Media", August 7, 2017, Computer Science [Overview of the project] [Problems that the invention aims to solve]

[0005] Using early fusion reduces computational complexity compared to late fusion, which offers higher accuracy but is computationally intensive. Late fusion, on the other hand, integrates data after neural network inference has been performed.

[0006] When using Early Fusion, it is necessary to ensure that multiple input data are the same size. The size of an input data can be represented by its channels, height, and width. Specifically, ensuring that multiple input data are the same size means that at least two of the three dimensions (channels, height, and width) are the same for each input data. Hereafter, these three dimensions will be referred to as "dimensions."

[0007] For example, when given multiple input data points that differ in both height and width, or in either of these, it may be necessary to enlarge or reduce the input data to make the sizes of the multiple input data points the same.

[0008] Consequently, when using Early Fusion, which is supposed to reduce computational complexity, expanding the input data can add redundant information to the input data, reducing the effectiveness of the computational complexity reduction. Conversely, shrinking the input data can result in information loss, leading to a decrease in inference accuracy.

[0009] Furthermore, Non-Patent Document 1 proposes multimodal fusion methods, including Early fusion and Late fusion, as well as Joint fusion and Common Space fusion.

[0010] The present invention aims to provide an inference device, inference method, and inference program that can suppress the reduction in computational load and the decrease in inference accuracy, even when using Early Fusion. [Means for solving the problem]

[0011] The inference apparatus according to this disclosure includes input size identification means for identifying the size of each of a plurality of input data; reference size determination means for determining a reference size; input data deformation means for deforming the input data based on the reference size to generate a plurality of deformed data; data deformation means for combining the deformed data into a single data; and inference means for performing inference with a single data as input.

[0012] The inference method according to the present disclosure specifies the size of each of a plurality of input data, determines a reference size, transforms the input data based on the reference size to generate a plurality of transformed data, combines the transformed data into one data, and performs inference using the one data as an input.

[0013] The inference program according to the present disclosure causes a computer to specify the size of each of a plurality of input data, determine a reference size, transform the input data based on the reference size to generate a plurality of transformed data, combine the transformed data into one data, and perform inference using the one data as an input.

Advantages of the Invention

[0014] In the present disclosure, even when Early fusion is used, the reduction effect of the computation amount is suppressed from decreasing, and the decrease in the accuracy of inference is also suppressed.

Brief Description of the Drawings

[0015] [Figure 1] It is a block diagram showing an example of the configuration of an inference device. [Figure 2] It is an explanatory diagram showing an example of processing such as folding of input data. [Figure 3] It is an explanatory diagram showing an example of a method for determining a reference size. [Figure 4] It is an explanatory diagram showing an example of a method for determining a reference size. [Figure 5] It is an explanatory diagram showing an example of transformation of input data. [Figure 6] It is an explanatory diagram showing an example of transformation processing. [Figure 7] It is an explanatory diagram for explaining a specific example of transformation processing. [Figure 8] It is an explanatory diagram showing another example of transformation processing. [Figure 9] It is an explanatory diagram showing an example of folding input data in directions related to two dimensions. [Figure 10]This is an explanatory diagram to provide a more detailed explanation of folding the input data in two dimensions. [Figure 11] This is a flowchart showing the operation of the inference device. [Figure 12] This block shows another example of the configuration of an inference device. [Figure 13] This is a flowchart showing the operation of the search unit in the inference device. [Figure 14] This is a block diagram showing an example of the configuration of an information processing system. [Figure 15] This is a block diagram showing the main components of the inference device. [Modes for carrying out the invention]

[0016] The embodiments will be described below with reference to the drawings.

[0017] Embodiment 1. Figure 1 is a block diagram showing an example of the configuration of an inference device. The inference device 100 shown in Figure 1 includes an input size identification unit 101, a reference size calculation unit 102, an input data transformation unit 103, a data merging unit 104, and an inference unit 105.

[0018] The input size identification unit 101 identifies the size (input size) of each of the multiple input data. The reference size calculation unit 102 determines the reference size, which is the size of a reference two-dimensional plane, based on the input sizes. The two-dimensional plane is defined by height and width.

[0019] Figure 1 shows two input data (input data A and input data B) as an example, but three or more types of input data may be input to the inference device 100 (specifically, the input size identification unit 101).

[0020] The input data transformation unit 103 and the data merging unit 104 transform the input data so that its size becomes a reference size, and then merge multiple transformed input data in the channel direction. Specifically, the input data transformation unit 103 resizes the input data so that its two-dimensional plane becomes a reference size, and then performs a transformation process that folds it in the channel direction. The data merging unit 104 merges multiple transformed input data in the channel direction.

[0021] As will be described later, the input data transformation unit 103 transforms the input data by, for example, dividing (splitting) the input data into multiple data and folding it. The data merging unit 104 merges the transformed input data, that is, the multiple data obtained by the transformation, in the channel direction. The data merging unit 104 generates one merged data from the multiple data obtained by the transformation. This merged data is sometimes referred to as input data. This input data is the input data to the inference unit 105, and although the representation is the same, it is different from the input data to the inference device 100.

[0022] The inference unit 105 includes an inference model. The inference unit 105 provides the inference model with one input data (one combined data) and obtains an inference result. If the inference model is a convolutional neural network, the number of channels in the first layer (e.g., a convolutional layer) is equal to the number of channels in the input data.

[0023] Next, an example of the transformation process performed by the input data transformation unit 103, namely the folding of the input data, will be explained with reference to Figure 2.

[0024] In the following, we will use color image data (hereinafter referred to as "color image") as input data A and monochrome image data (hereinafter referred to as "monochrome image") as input data B as an example. That is, input data A and input data B are both images, but their formats are different. However, input data A and input data B may be in the same format (in this example, either a color image or a monochrome image). Furthermore, the input data to the inference device 100 is not limited to image data. For example, the input data may be audio data, text data, wireless signals, etc.

[0025] When input data A, which is a color image, and input data B, which is a monochrome image, are input, the reference size calculation unit 102 determines the reference size based on the input size identified by the input size identification unit 101. The input data transformation unit 103 folds input data A and input data B in the channel direction. Specifically, the input data transformation unit 103 divides input data A and input data B by the reference size and folds them in the channel direction. As described above, the division and folding of input data is an example of input data transformation.

[0026] The data merging unit 104 combines the transformed data in the channel direction. In the example shown in Figure 2, input data A includes R data, G data, and B data, each of which is split into two, resulting in 6-channel data after transformation. Similarly, input data B is split into two, resulting in 2-channel data after transformation. These two data sets are combined in the channel direction. The combined data has 8 channels.

[0027] Next, we will explain the folding and other processing methods using specific examples.

[0028] Figures 3 and 4 are explanatory diagrams illustrating an example of a method for determining the standard size. Hereafter, the number of channels, height, and width of the input data will be represented as [number of channels, height, width]. Similarly, the height and width of the standard size will be represented as [height, width].

[0029] In the example shown in Figure 3, the number of channels, height, and width of input data A are [3,210,80]. The number of channels, height, and width of input data B are [1,100,160]. In this case, the reference size calculation unit 102 determines the reference size (height, width) to be [100,80].

[0030] In other words, the reference size calculation unit 102 determines the minimum value for the height and width of each of the multiple input data as the height and width of the reference size (reference size).

[0031] Figure 5 is an explanatory diagram showing an example of input data transformation.

[0032] In the example shown in Figure 5, the number of channels, height, and width of input data A are [3,200,80]. The number of channels, height, and width of input data B are [1,100,160]. In this case, the input data transformation unit 103 transforms each input data so that the two-dimensional plane of each data conforms to the reference size. Then, for each input data, the input data transformation unit 103 folds the transformed data (for example, multiple data obtained by dividing the input data) in the channel direction.

[0033] Therefore, in the example shown in Figure 5, 6-channel 2D plane data is generated from 3-channel input data A. Also, 2-channel 2D plane data is generated from 1-channel input data B.

[0034] Figure 5 shows an example where input data A is divided by a standard size in the height direction and then folded in the channel direction. Similarly, it shows an example where input data B is divided by a standard size in the width direction and then folded in the channel direction.

[0035] In the example shown in Figure 5, the sizes of input data A and input data B are multiples of the standard size, but the size of input data is not necessarily a multiple of the standard size. If the size of input data is not a multiple of the standard size, that is, if the size of input data has a fractional part relative to the standard size, the input data transformation unit 103 adjusts the size of input data so that it becomes a multiple of the standard size.

[0036] The following methods can be considered for adjustment.

[0037] First, the input data transformation unit 103 calculates the fractional part for both the height and width of the input data. For example, if the base size is [100, 80], and the number of channels, height, and width of the input data are [3, 210, 80], then the fractional part in the height direction is 210% 100 = 10. The fractional part in the width direction is 80% 80 = 0. Note that % is used as the operator for the modulo operation (the operation to find the remainder of a division).

[0038] Then, when the fractional part is less than or equal to a predetermined threshold, the input data transformation unit 103 reduces the input data so that the size becomes a multiple of the standard size. Figure 6 shows an example where the height of input data A is 210 and is reduced to 200. The threshold is, for example, half of the standard size, but the user may arbitrarily determine the threshold. The input data transformation unit 103 achieves size reduction by, for example, downsampling pixels in the input data, but the size may also be reduced by trimming the input data.

[0039] When the fractional part is greater than a predetermined threshold, the input data transformation unit 103 expands the size of the input data to a multiple of the reference size, provided that it does not exceed the original size. For example, if the reference size is [100,80], and the number of channels, height, and width of the input data are [3,210,80], the height will be expanded to 300. The input data transformation unit 103 achieves the expansion of the size by, for example, interpolating the pixels of the input data, but it may also increase the size by adding zero data.

[0040] In summary, when the input data transformation unit 103 divides the input data into a standard size, if the size of the input data results in a fraction of the standard size, it performs an adjustment process to resize the input data before dividing it. This adjustment process can be applied regardless of whether the input data transformation unit 103 uses transformation method 1-A, transformation method 1-A, or transformation method 2, which will be described later.

[0041] Furthermore, the processing after resizing or resizing (deformation and folding) is the same as the processing shown in Figure 5.

[0042] Figure 7 is an explanatory diagram illustrating an example of a transformation process. The above transformation process will be referred to as transformation method 1-A. As shown in Figure 7, in transformation method 1-A, the input data is divided by a standard size in the height or width direction and folded in the channel direction. However, a transformation process may also be performed in which the pixels of the input data are folded in a way that tiled them sequentially in the channel direction. This transformation method will be referred to as transformation method 1-B.

[0043] In Figure 7, the numbers within the rectangles correspond to the pixel numbers. The reference width (corresponding to the number of pixels if the input data is an image) is the width of the reference size. Figure 7 also shows an example of transformation method 1-A where the input data is divided and folded in the width direction by the reference size, but the same processing applies when the input data is divided and folded in the height direction by the reference size.

[0044] The following transformation process can also be performed. Let's call the following transformation method 2.

[0045] As shown by the dashed lines in Figure 8, multiple frames of a standard size are set for the input data. These frames overlap. Figure 8 shows an example where two frames are set. Note that Figure 8 also illustrates a case where the size of the input data is not a multiple of the standard size.

[0046] The input data transformation unit 103 performs the transformation process, including the overlapping regions. Specifically, the transformation process is performed as follows:

[0047] Let w be the width of the original image, and t be the width of the reference size. Also, let o be the width of the overlapping area (overlap width). The overlap width can be set arbitrarily. Then, the input data transformation unit 103 calculates the number of divisions n in the width direction.

[0048] The number of divisions n is the value that minimizes the difference between the width {tn-o(n-1)} just before folding and w, as expressed in Equation 1.

[0049]

number

[0050] The input data transformation unit 103 resizes (expands or shrinks) the input data with width w to {tn-o(n-1)}. Then, the input data is folded to the standard size so that it overlaps with the width o. Note that if w = {tn-o(n-1)}, the input data transformation unit 103 does not resize the input data.

[0051] In other words, the input data transformation unit 103 extracts multiple regions of a standard size from the input data, but if the size of the input data has a fractional part relative to the standard size, it performs an adjustment process to resize the input data to eliminate the fractional part.

[0052] Figure 8 shows Example 1, where the reference width is 4 and the overlap width o is 1. In Example 1, n=2. In Example 1, since w>{tn-o(n-1)}, the input data is reduced in size. Also, since the number of divisions n=2, the input data is divided into 2 parts. Note that in the folded data, pixel 4' is an overlapping pixel.

[0053] Furthermore, Example 2 is shown where the reference width is 2 and the overlap width o is 1. In Example 2, n=7. Since the number of divisions n=7, the input data is divided into 7 parts. Pixels with pixel numbers 2 to 7 are overlapping pixels.

[0054] The above explanation primarily used the example of input data being folded in the width direction. The same concepts can be applied to cases where input data is folded in the height direction as well.

[0055] It is also possible to transform the input data in both the width and height directions, in which case either transformation method 1-A or transformation method 1-B described above may be used for either direction. Furthermore, the transformation method for the width direction and the transformation method for the height direction may be different. For example, transformation method 1-A or transformation method 1-B described above may be used for the width direction, and the above transformation method may be used for the height direction. Alternatively, transformation method 2 described above may be used for the width direction, and transformation method 1-A or transformation method 1-B described above may be used for the height direction.

[0056] Furthermore, when resizing input data, the input data transformation section 103 may be uniformly enlarged or reduced in both the width and height directions. Also, the overlap width may be the same in both the width and height directions, or it may be different in both directions.

[0057] In the above explanation, the input data was divided by a standard size in the height or width direction and folded in the channel direction. That is, the data divided by one dimension in the input data (divided data) was folded in the direction of another dimension (specifically, the channel). However, it is also possible to fold the input data, which is divided by one dimension, in the direction of multiple other dimensions. The following transformation process will be called transformation method 3.

[0058] Figure 9 is an explanatory diagram illustrating an example of folding input data in two dimensions. Figure 9 shows an example where input data B, a monochrome image, is divided in the width direction and then folded in the height and channel directions, with input data A, a color image, as the base size.

[0059] Specifically, input data B is split into data (in this example, images) a-d, data a and b are folded in the height direction, and data c and d are folded in the height direction. Furthermore, the combined data consisting of data a and b, and the combined data consisting of data c and d are folded in the channel direction. Also, the order of folding in the height direction and the channel direction is not fixed; the channel direction can be done first.

[0060] Furthermore, when folding the data (image) in the height direction, margins may be added to data a to d to avoid interference with adjacent pixels.

[0061] Hereafter, the folding of data (images) in a certain direction may be described as data being arranged. In other words, dividing (splitting) data into multiple data and arranging them in a certain dimension may be described as data arrangement. The data division (splitting) and data arrangement processes are performed by the data transformation unit 103.

[0062] Figure 10 is an explanatory diagram that provides a more detailed explanation of folding (splitting and arranging) of input data in two dimensions. Figure 10 also shows an example in which input data B is split in the width direction, and the resulting data (images) a-d are arranged in the height direction and channel direction.

[0063] Since the dimension related to folding (the direction of folding) is the width direction, the minimum value of the width of each input data is taken as the width of the base size. Also, since the input data is folded in the height direction, the maximum value of the height of each input data is taken as the height of the base size.

[0064] Let's take [3,210,40] as an example for the number of channels, height, and width of input data A. Let's take [1,100,160] as an example for the number of channels, height, and width of input data b.

[0065] The standard height and width are t h t w Let h and w be the height and width of the input data to be folded. Also, set the margin width m. The margin width m is set by the user, for example.

[0066] The input data transformation unit 103 calculates the number of times (number of divisions) the input data to be folded (input data B in the example shown in Figure 10) is folded in the height direction. h We seek.

[0067] n times h As shown in equation 2, {t h -{hn h +m(n h This is the value that minimizes (-1). In the example shown in Figure 10, n h = 2

[0068]

number

[0069] Further, the input data transformation unit 103 determines the number of times n that the input data to be folded (in the example shown in FIG. 10, input data B) is folded in the channel direction. c to obtain.

[0070] The number of times n c is, as represented by Equation 3, the value that minimizes {w - t w n h n c}. In the example shown in FIG. 10, n c = 2.

[0071]

Equation

[0072] Furthermore, the input data transformation unit 103 enlarges or reduces the input data so that the height h and width w of the input data to be folded (in the example shown in FIG. 10, input data B) become the following values. The input data transformation unit 103 may perform trimming or zero setting.

[0073] h = {t h - m(n h - 1)} / n h w = t w n h n c

[0074] The input data transformation unit 103 divides the input data into (n h × n c ) in the width direction and folds it in the channel direction. In the example shown in FIG. 10, (n h × n c ) = 4. The input data transformation unit 103 arranges the plurality of data obtained by the division in a state where a margin is provided between the data. Note that the input data transformation unit 103 sets, for example, 0 in the margin portion.

[0075] Next, referring to the flowchart of FIG. 11, the operation of the inference device 100 will be described.

[0076] The input size identification unit 101 identifies the size (input size) of each of the multiple input data (step S101). The reference size calculation unit 102 determines the reference size, which is the size of a reference two-dimensional plane, based on the input sizes (step S102). The method for determining the reference size is as described above.

[0077] The input data transformation unit 103 transforms each input data so that the two-dimensional plane of the input data becomes the standard size (step S103). That is, after enlarging or shrinking the input data, the input data transformation unit 103 divides the input data with respect to one or more dimensions and folds it in the direction of a dimension other than the one or more dimensions. Note that the input data transformation unit 103 may not enlarge or shrink the input data.

[0078] The data merging unit 104 combines multiple data obtained by transformation (e.g., resizing and folding) to obtain a single data (step S104).

[0079] The inference unit 105 uses machine learning, such as a neural network, to obtain prediction results for the input data (step S105).

[0080] As described above, in this embodiment, the inference device 100 transforms each input data to match the size based on a reference size, and then combines the multiple input data. Therefore, when using Early fusion, loss of information is suppressed, and thus the decrease in inference accuracy can be suppressed. Furthermore, the effect of reducing computational complexity by Early fusion is not diminished.

[0081] Therefore, the inference device 100 of this embodiment can be used, for example, to run efficient machine learning applications while maintaining accuracy in environments with limited computing resources.

[0082] Embodiment 2. Figure 12 is a block diagram showing another example of the configuration of an inference device. The inference device 200 shown in Figure 12 includes an input size identification unit 101, a reference size calculation unit 102, an input data transformation unit 103, a data merging unit 104, and an inference unit 105, in addition to a search unit 106.

[0083] The configuration and functions of the input size identification unit 101, the reference size calculation unit 102, the input data transformation unit 103, the data merging unit 104, and the inference unit 105 are the same as those in the first embodiment. The search unit 106 searches for parameters to be used by the input data transformation unit 103.

[0084] The search unit 106 supplies usable parameters to the input data transformation unit 103. The input data transformation unit 103 uses the supplied parameters to perform processing similar to that in the first embodiment. Referring to the first embodiment described above, the parameters usable by the input data transformation unit 103 include, for example, the transformation methods (transformation method 1-A, transformation method 1-B, transformation method 2), the overlap width value in modification method 2, and the margin width value. However, the parameters are not limited to these, and if the input data transformation unit 103 is configured to use other parameters, other parameters may also be included in the search targets.

[0085] The search unit 106 sequentially supplies different parameter combinations to the input data transformation unit 103. The parameter combinations are, for example, transformation method 1-A, transformation method 2 and overlap width = 10, transformation method 3 and margin width = 10.

[0086] When any combination of parameters is supplied to the input data transformation unit 103, the input data transformation unit 103 and the data merging unit 104 perform the same processing as in the first embodiment and output the merged data to the inference unit 105. The inference unit 105 uses the inference model to obtain prediction results for the input data.

[0087] The search unit 106 optimizes the parameters and the neural architecture of the inference model in the inference unit 105 using the Neural Architecture Search (NAS) method. For example, the search unit 106 receives the prediction results from the inference unit 105. The search unit 106 then uses the loss of the prediction results as the objective function and updates the parameters and neural architecture to minimize the value of the objective function.

[0088] When the inference device 200 is in actual operation, the search unit 106 is excluded from the inference device 200. In other words, the inference device 200 is used in the same manner as the inference device 100 in the first embodiment.

[0089] Next, the operation of the search unit 106 in the inference device 200 will be explained with reference to the flowchart in Figure 13.

[0090] The search unit 106 randomly selects the deformation method, overlap width, margin width, and neural architecture from multiple candidates (step S201).

[0091] The search unit 106 supplies the selected deformation method, overlap width, and margin width to the input data deformation unit 103. The input data deformation unit 103 performs the same processing as in the first embodiment using the supplied parameters. The search unit 106 also supplies the selected neural architecture to the inference unit 105. The inference unit 105 performs inference using the inference model composed of the supplied neural architecture.

[0092] The search unit 106 inputs a pre-prepared training dataset to the input size determination unit 101 and the input data transformation unit 103. The parts of the inference device 200 excluding the search unit 106 perform the same processing as in the first embodiment, and the inference unit 105 obtains the prediction result (step S202).

[0093] The search unit 106 uses the loss of the prediction result as the objective function and updates the parameters and neural architecture to minimize the objective function (step S203).

[0094] The search unit 106 checks whether the processes in steps S202 and S203 have been executed a predetermined number of times (step S204). If the number of executions has not reached the predetermined number, the process returns to step S202. If the number of executions has reached the predetermined number, the process terminates.

[0095] In this embodiment, in addition to the effects of the first embodiment, the effect of being able to determine the optimal parameters and neural architecture can be obtained.

[0096] Furthermore, while each of the above embodiments can be implemented using hardware, they can also be realized using a computer equipped with a processor such as a CPU (Central Processing Unit) and memory.

[0097] For example, a program for performing the method (processing) in the above embodiment may be stored in a storage device (storage medium), and each function may be realized by executing the program stored in the storage device using the CPU.

[0098] Figure 14 is a block diagram showing an example of a computer having a CPU. The computer is implemented in inference devices 100 and 200. The CPU 1001 realizes each function in the above embodiment by executing processing according to the program (software element: code) stored in the storage medium 1003. Specifically, it realizes the functions of the input size determination unit 101, the reference size calculation unit 102, the input data transformation unit 103, the data merging unit 104, and the inference unit 105 in the inference device 100 shown in Figure 1. It also realizes the functions of the input size determination unit 101, the reference size calculation unit 102, the input data transformation unit 103, the data merging unit 104, the inference unit 105, and the search unit 106 in the inference device 200 shown in Figure 12.

[0099] Multiple processors (computers) can work together to implement the functions of inference units 100 and 200. Alternatively, a CPU and a GPU (Graphics Processing Unit) can work together to implement the functions of inference units 100 and 200.

[0100] The storage medium 1003 is, for example, a non-transitory computer-readable medium. Non-transitory computer-readable media include various types of tangible storage media. Specific examples of non-transitory computer-readable media include magnetic recording media (e.g., hard disks), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Compact Disc-Read Only Memory), CD-Rs (Compact Disc-Recordable), CD-R / Ws (Compact Disc-ReWritable), and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs).

[0101] Furthermore, the program may be stored in various types of transient computer-readable medium. The transient computer-readable medium may be supplied with the program, for example, via a wired or wireless communication channel, i.e., via electrical signals, optical signals, or electromagnetic waves.

[0102] Memory 1002 is implemented, for example, as RAM (Random Access Memory) and is a storage means that temporarily stores data when the CPU 1001 executes processing. It is also conceivable that a program held by the storage medium 1003 or a temporary computer-readable medium is transferred to memory 1002, and the CPU 1001 executes processing based on the program in memory 1002. Note that the storage medium 1003 and memory 1002 may be a single unit.

[0103] Figure 15 is a block diagram showing the main parts of the inference device. The inference device 10 shown in Figure 15 includes an input size identification means 11 (implemented in an input size identification unit 101 in an embodiment) that identifies the size of each of a plurality of input data, a reference size determination means 12 (implemented in an embodiment in a reference size calculation unit 102) that determines a reference size, an input data deformation means 13 (implemented in an embodiment in an input data deformation unit 103) that deforms the input data based on the reference size and generates a plurality of deformed data, a data merging means 14 (implemented in an embodiment in a data merging unit 104) that combines the deformed data into a single data, and an inference means 15 (implemented in an embodiment in an inference unit 105) that performs inference using a single data as input.

[0104] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0105] (Appendix 1) An input size identification means for identifying the size of each of the multiple input data, A means for determining the standard size, An input data transformation means that transforms the input data based on the aforementioned reference size to generate a plurality of transformed data, A data merging means that combines the transformed data into a single data, An inference means that performs inference using the aforementioned one data as input. An inference device equipped with the following features.

[0106] (Note 2) The data transformation means is The input data is resized to the reference size in one or more dimensions (e.g., height and width). The resized data is arranged in a direction of a dimension other than the aforementioned one or more dimensions (for example, the channel direction). The inference device described in Appendix 1.

[0107] (Note 3) The data transformation means is Multiple regions of the standard size are extracted from the input data, and the extracted regions include regions that overlap with adjacent regions. If the size of the input data has a fractional part relative to the base size, the input data is resized to eliminate the fractional part when the input data is transformed. The inference device described in Appendix 1.

[0108] (Note 4) The data transformation means combines the transformed data into a single data by providing a margin between the multiple transformed data. The inference device described in Appendix 1.

[0109] (Note 5) The data transformation means includes a search means (implemented by the search unit 106 in this embodiment) that searches for the optimal values ​​of the parameters used during transformation (e.g., transformation method, overlap width, margin width) and the neural architecture of the inference model included in the inference means. An inference device as described in any of the appendices 1 through 4.

[0110] (Note 6) Identify the size of each of the multiple input data, Determine the standard size, The input data is transformed based on the aforementioned reference size to generate multiple transformed data sets. Combine the transformed data into a single data set. The aforementioned single data point is used as input for inference. Reasoning method.

[0111] (Note 7) When transforming the input data, the input data is resized to the standard size for one or more dimensions. The resized data is arranged in a direction of a dimension other than the one or more dimensions mentioned above. The inference method described in Appendix 6.

[0112] (Note 8) When transforming the input data, multiple regions of the standard size are extracted from the input data, and the extracted regions include regions that overlap with adjacent regions. If the size of the input data has a fractional part relative to the base size, the input data will be resized to eliminate the fractional part when the input data is transformed. The inference method described in Appendix 6.

[0113] (Note 9) Add a margin between the multiple transformed data and combine the transformed data into a single data. The inference method described in Appendix 6.

[0114] (Note 10) Explore the optimal values ​​of the parameters used during deformation and the neural architecture of the inference model used for inference. The inference method described in any of the appendices 6 to 9.

[0115] (Note 11) To the computer, Specify the size of each of the multiple input data, Determine the standard size, The input data is transformed based on the aforementioned reference size to generate multiple transformed data sets. Combine the transformed data into a single data set. The aforementioned single data point is used as input to perform inference. An inference program for that purpose.

[0116] (Note 12) To the computer, When transforming the input data, the input data is resized to the standard size for one or more dimensions. The resized data is arranged in a direction that is different from the one or more dimensions mentioned above. The inference program described in Appendix 11.

[0117] (Note 13) To the computer, When transforming the input data, multiple regions of the aforementioned standard size are extracted from the input data, and the extracted regions include regions that overlap with adjacent regions. If the size of the input data has a fractional part relative to the base size, the input data will be resized to eliminate the fractional part when the input data is transformed. The inference program described in Appendix 11.

[0118] (Note 14) A margin is added between the multiple transformed data points, and the transformed data is combined into a single data point. The inference program described in Appendix 11.

[0119] (Note 15) This process explores the optimal values ​​for the parameters used during deformation and the neural architecture of the inference model used for inference. An inference program described in any of the appendices 11 to 14.

[0120] Some or all of the configurations described in Appendices 2 to 5, which are dependent on Appendice 1 above, may be applied to various hardware, software, various recording means for recording software, or systems, provided that they do not deviate from the embodiments described above. [Explanation of symbols]

[0121] 10,100,200 Reasoning device 11 Input size determination means 12. Means for determining the reference size 13 Input data transformation means 14. Data linking means 15 Means of reasoning 101 Input size identification section 102 Reference Size Calculation Unit 103 Input Data Transformation Section 104 Data merging section 105 Reasoning part 106 Search Department 1001 CPU 1002 memory 1003 Storage medium

Claims

1. An input size identification means for identifying the size of each of multiple input data, A means for determining the standard size, An input data transformation means that transforms the input data based on the aforementioned reference size to generate a plurality of transformed data, A data merging means that combines the transformed data into a single data, An inference means that performs inference using the aforementioned one data as input. An inference device equipped with the following features.

2. The data transformation means is The input data is resized to the standard size for one or more dimensions. The resized data is arranged in a direction of a dimension other than the one or more dimensions mentioned above. The inference device according to claim 1.

3. The data transformation means is Multiple regions of the standard size are extracted from the input data, and the extracted regions include regions that overlap with adjacent regions. If the size of the input data has a fractional part relative to the base size, the input data is resized to eliminate the fractional part when the input data is transformed. The inference device according to claim 1.

4. The data transformation means combines the transformed data into a single data by adding a margin between the multiple transformed data. The inference device according to claim 1.

5. The data transformation means includes a search means for the optimal values ​​of the parameters used during transformation and the neural architecture of the inference model included in the inference means. An inference device according to any one of claims 1 to 4.

6. Identify the size of each of the multiple input data, Determine the standard size, The input data is transformed based on the aforementioned reference size to generate multiple transformed data sets. Combine the transformed data into a single data set. The aforementioned single data point is used as input for inference. Reasoning method.

7. When transforming the input data, the input data is resized to the standard size for one or more dimensions. The resized data is arranged in a direction of a dimension other than the one or more dimensions mentioned above. The inference method according to claim 6.

8. When transforming the input data, multiple regions of the aforementioned standard size are extracted from the input data, and the extracted regions include regions that overlap with adjacent regions. If the size of the input data has a fractional part relative to the base size, the input data will be resized to eliminate the fractional part when the input data is transformed. The inference method according to claim 6.

9. A margin is added between the multiple transformed data points, and the transformed data is combined into a single data point. The inference method according to claim 6.

10. On the computer, Specify the size of each of the multiple input data, Determine the standard size, The input data is transformed based on the aforementioned reference size to generate multiple transformed data sets. Combine the transformed data into a single data set. The aforementioned single data point is used as input to perform inference. An inference program for that purpose.

Images