Information processing device, information processing system, information processing method, and program

Abstract

In order to improve inference processing throughput, an information processing device according to the present invention comprises: a data configuration means that generates configuration data comprising a plurality of sets of data, at least a portion of each set of data including an inference target; a configuration data inference means that performs inference on a target included in the configuration data; a data inference means that performs inference on a target included in at least some data; and a target inference means that determines, for each set of data constituting the configuration data, whether the inference on the target included in the configuration data is successful or unsuccessful, performs inference on a target included in a set of data for which the inference on the target included in the configuration data is successful on the basis of the configuration data and the result of performing the inference on the target included in the configuration data, and performs inference on a target included in a set of data for which the inference on the target included in the configuration data is unsuccessful, on the basis of the result of performing inference on a target included in data.

Description

[Technical field]

[0001] The present invention relates to inference processing using machine learning. [Background technology]

[0002] Non-Patent Documents 1 to 4 disclose techniques related to image inference processing using deep learning. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, “Deep Residual Learning for Image Recognition”, [online], 10 December 2-15, Cornel University, [Retrieved October 13, 2021], Internet<URL: https: / / arxiv.org / abs / 1512.03385> [Non-Patent Document 2] Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", [online], 6 January 2016, Cornel University, [Retrieved October 13, 2021], Internet<URL:https: / / arxiv.org / abs / 1506.01497> [Non-Patent Document 3] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C. Berg, "SSD: Single Shot MultiBox Detector", [online], 29 December 2016, Cornel University, [Retrieved October 13, 2021, Internet,<URL:https: / / arxiv.org / abs / 1512.02325> [Non-Patent Document 4] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollar, "Focal Loss for Dense Object Detection", [online], 7 February 2018, Cornel University, [Retrieved October 13, 2021], Internet,<URL:https: / / arxiv.org / abs / 1708.02002> Summary of the Invention [Problem to be solved by the invention]

[0004] Non-Patent Documents 1 to 4 do not disclose any technology related to improving the throughput of inference processing. An object of the present invention is to provide an information processing device or the like that improves the throughput of inference processing. [Means for solving the problem]

[0005] In one aspect of the present invention, an apparatus comprises: A data construction means for generating constructed data constructed using a plurality of data, at least a part of which includes an inference target; A configuration data inference means for executing inference on an object included in the configuration data; A data inference means for performing inferences on objects contained in at least a portion of the data; For each of the data constituting the configuration data, it is determined whether the inference for the object included in the configuration data is successful or unsuccessful; As an inference for an object included in the data in which an inference for an object included in the configuration data is successful, an inference for the object is performed based on the inference result for the object included in the configuration data and the configuration data; As an inference for an object contained in data for which inference for an object contained in the configuration data has failed, inference for the object is performed based on the inference results for the object contained in the data. Object inference means and Includes.

[0006] In one aspect of the present invention, a system includes: The information processing device; A data acquisition device that outputs a plurality of data including at least a part of the target to an information processing device; A recognition device that receives an inference result of an object from the information processing device and performs recognition related to the object based on the inference result that has been received; Includes.

[0007] An information processing method according to one aspect of the present invention includes: generating constructed data including a plurality of pieces of data at least a part of which includes a subject of inference; Perform inference on objects contained in the configuration data; Performing inferences on at least some of the data about objects contained in the data; For each of the data constituting the configuration data, it is determined whether the inference for the object included in the configuration data is successful or unsuccessful; As an inference for an object included in the data in which an inference for an object included in the configuration data is successful, an inference for the object is performed based on the inference result for the object included in the configuration data and the configuration data; As an inference for an object included in data in which an inference for an object included in the configuration data has failed, an inference for the object is performed based on the inference result for the object included in the data.

[0008] An information processing method according to one aspect of the present invention includes: An information processing device executes the above method, The data acquisition device outputs a plurality of data at least a part of which includes the target to the information processing device; The recognition device acquires an inference result for the object from the information processing device, and performs recognition related to the object based on the acquired inference result.

[0009] A recording medium according to one embodiment of the present invention includes: A process of generating constituted data composed of a plurality of pieces of data at least a part of which includes a subject of inference; A process of performing inference on objects contained in the configuration data; performing inferences on at least some of the data about objects contained in the data; For each of the data constituting the configuration data, it is determined whether the inference for the object included in the configuration data is successful or unsuccessful; As an inference for an object included in the data in which an inference for an object included in the configuration data is successful, an inference for the object is performed based on the inference result for the object included in the configuration data and the configuration data; A process of performing an inference on an object based on the inference result for the object included in the data as an inference on an object included in the configuration data that has failed to be inferred; A program that causes a computer to execute the above is recorded. Effect of the Invention

[0010] According to the present invention, the throughput of the inference process can be improved. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an information processing device according to a first embodiment of the present invention. [Diagram 2] FIG. 13 is a diagram illustrating an example of the relationship between CNN size and utilization efficiency of computational resources. [Diagram 3]FIG. 1 is a schematic diagram showing an example of a NN used in an image classification task. [Figure 4] FIG. 1 is a schematic diagram showing an example of a neural network used in an object detection task. [Diagram 5] FIG. 11 is a block diagram showing an example of a configuration of an information processing device according to a second embodiment. [Figure 6] FIG. 13 is a diagram showing an example of a constituent image. [Figure 7] FIG. 11 is a flow diagram showing an example of an operation for generating a trained model in an information processing device according to the second embodiment. [Figure 8] FIG. 11 is a flow diagram showing an example of an operation of executing inference on a target in the information processing device according to the second embodiment. [Figure 9] FIG. 13 is a block diagram showing a configuration of an example of a variation of an information processing device according to the second embodiment. [Figure 10] FIG. 13 is a block diagram illustrating an example of a configuration of an information processing device according to a third embodiment. [Figure 11] FIG. 2 is a block diagram showing an example of a hardware configuration of an information processing device. [Figure 12] FIG. 1 is a block diagram showing an example of a configuration of an information processing system including an information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, each embodiment of the present invention will be described with reference to the drawings. Each drawing is for explaining each embodiment. However, each embodiment is not limited to the description of each drawing. In addition, each embodiment can be appropriately combined.

[0013] First Embodiment FIG. 1 is a block diagram showing an example of the configuration of an information processing device 101 according to a first embodiment of the present invention. The information processing device 101 includes an object inference unit 11, a data configuration unit 21, a configuration data inference unit 31, and a data inference unit 41. The data configuration unit 21 generates configured data configured using a plurality of data, at least a part of which includes an inference target. The configuration data inference unit 31 executes inference for an object included in the configured data. The data inference unit 41 executes inference for an object included in at least a part of the data. The object inference unit 11 determines whether inference for an object included in the configured data has succeeded or failed for each piece of data constituting the configured data. Then, the object inference unit 11 executes inference for an object included in data in which inference for an object included in the configured data has succeeded, based on the inference result for the configured data and the configured data. Then, the object inference unit 11 executes inference for an object included in data in which inference for an object included in the configured data has failed, based on the inference result for the data.

[0014] The data construction unit 21 uses a plurality of data including an inference target to generate constructed data that is data larger in scale than the data. The constructed data inference unit 31 executes inference on the target included in the constructed data. For data for which inference on the target has been successful in the constructed data inference unit 31, the target inference unit 11 executes inference on the target based on the inference result on the target included in the constructed data, which is data larger in scale than the data, and the constructed data. In other words, the information processing device 101 executes inference on the target based on the inference result using the constructed data, which is data larger in scale than the data including the target. Next, the relationship between the size of data and the improvement of the throughput of inference processing will be described.

[0015] Image inference processing using machine learning, such as image classification tasks and object detection tasks, is a high-load process. Therefore, various types of HW have been developed as hardware (HW) for performing such inference processing at high speed. For example, such HW includes a graphics processing unit (GPU) or a field programmable gate array (FPGA). Alternatively, such HW includes an application specific integrated circuit (ASIC) or an artificial intelligence (AI) chip. In the following description, HW for performing these inference processes is collectively referred to as an "AI device."

[0016] In order to realize high-speed and high-throughput inference processing, these AI devices are equipped with many arithmetic units, divide an input image, and process each divided image in parallel in different arithmetic units. Alternatively, the AI ​​device performs inference processing by connecting multiple arithmetic units with different functions in a pipeline manner. Note that some chips called central processing units (CPUs) have many arithmetic units as arithmetic units inside, or arithmetic units for specific purposes such as matrix operations that are often used in machine learning. In other words, some CPUs have a arithmetic unit equivalent to an AI device. Therefore, in the following description, unless a special distinction is required, the AI ​​device includes a CPU having a arithmetic unit equivalent to the above-mentioned AI device.

[0017] AI devices can execute inference processing with high throughput. However, since AI devices achieve high throughput by operating many computing units in parallel, it is necessary to operate many computing units at a high operating rate. For example, when the size of an image is small, the AI ​​device cannot fully utilize the parallelism in many computing units, and therefore cannot achieve high throughput. Note that data that is the target of an AI device is not limited to two-dimensional data such as images. For example, an AI device may operate on three-dimensional data such as data acquired using a radar (Radio Detecting and Ranging (RADAR)) or a lidar (light detection and ranging (LiDAR)).

[0018] FIG. 2 is a diagram showing an example of the relationship between the size of a convolutional neural network (CNN) used in parallel and the utilization efficiency of computing resources, measured using a certain AI device. The utilization efficiency in FIG. 2 is a value obtained by dividing the computing performance obtained during actual inference processing by the maximum computing performance required from the computing resources of the AI ​​device. The unit of computing performance is, for example, gigaflops (Giga Floating-point Operations Per Second (GFLOPS)), but is not limited to this. As shown in FIG. 2, when the CNN size used is about 500, the utilization efficiency is about 70% of the maximum computing performance of the AI ​​device. However, when the CNN size used is 100 or less, the utilization efficiency is 20% or less of the maximum computing performance of the AI ​​device.

[0019] In this way, the higher the parallelism, the higher the utilization efficiency of the AI ​​device and the higher the throughput of the inference process. Inference using data including multiple targets and increasing the size of the data to be inferred are effective methods for increasing parallelism. Therefore, the information processing device 101 of the first embodiment generates configuration data configured using multiple data as large-sized data. Then, the information processing device 101 executes inference using the configuration data, thereby improving the parallelism in the AI ​​device and, as a result, improving the throughput of the inference process.

[0020] However, inference for an object included in constituted data configured using a plurality of data including an object may be more likely to fail in inference for some objects compared to inference for objects included in individual data. Therefore, the information processing device 101 includes a data inference unit 41. The data inference unit 41 executes inference for an object included in the data. Then, for data for which inference for an object has failed in constituted data inference unit 31, the object inference unit 11 executes inference for the object using the inference result of the data inference unit 41. Therefore, the information processing device 101 can execute inference for an object even for data for which inference for an object included in constituted data has failed.

[0021] <Second embodiment> [Description of related inference tasks] An inference task related to the description of the second embodiment will be described. One of the important tasks in image analysis using machine learning is an image classification task. The image classification task is a task to determine or classify the class of an object appearing in an image from a group of classes given in advance. In recent years, image classification tasks using deep learning, particularly machine learning, have been widely used. In the machine learning of the image classification task, a group of images for learning and class information of the object appearing in each image are given as correct answer data. Then, the image classification task acquires a trained model by machine learning using the group of images for learning and the class information. Then, the image classification task applies the trained model to an image including an object, infers the class of the object in the image, and outputs the inferred class.

[0022] In addition, an inference task such as an image classification task may output information related to inference, such as the confidence of the inference, in addition to the inference result. For example, an image classification task may output the confidence of the inference in addition to the class. In the following description, as an example, an inference task used in the description outputs a pair of an inference result and confidence. However, this does not limit the tasks used in this embodiment. Each task may not output confidence, or may output a value different from the confidence. Therefore, in the following description, a description of information related to inference, such as confidence, may be omitted unless particularly necessary. In addition, an image classification task may output multiple pairs of classes and confidence for one image.

[0023] An example of the configuration of an image classification task will be described with reference to the drawings. FIG. 3 is a diagram showing an example of a neural network (NN) used in an image classification task. The NN in FIG. 3 includes a feature extraction layer and a classification layer. FIG. 3 includes a "Residual Network (ResNet)" as an example of the feature extraction layer. ResNet is one of the NN models, and is a model that does not learn "the optimal output required in a certain layer" but learns "a residual function referring to the input of the layer". In FIG. 3, an image is first input to the feature extraction layer. The feature extraction layer calculates features related to the image and outputs the features to the classification layer. The classification layer outputs a pair of the class and certainty of the object in the image based on the features.

[0024] Another major task in image analysis using machine learning is the object detection task in an image. The object detection task is a task to generate a list of pairs of the position, class, and certainty of a detection target present in an image. Deep learning is also widely used for the object detection task. In the machine learning of the object detection task, a group of images for learning, the position of the detection target in each image, and the class of the detection target are given as correct answer data. Note that, although not limited to the following, the coordinates of the four vertices of a rectangular area in which the object is reflected (bounding box (BB)) are often used as the position of the target. Note that, in the following explanation, BB is also used as an example of the position of the detection target.

[0025] The object detection task then obtains a trained model through machine learning using a group of images for training, the position of the detection target in the image, and the class of the detection target. The object detection task then applies the trained model to an image including the detection target to infer the BB and class of the detection target in the image, and outputs a set of the inferred BB, class, and certainty of the detection target. If an image includes multiple targets, the object detection task outputs a set of the BB, class, and certainty of each of the multiple detection targets included in the image.

[0026] An example of an object detection task will be described with reference to the drawings. FIG. 4 is a diagram showing an example of a NN used in an object detection task. In addition to the configuration of the NN in FIG. 3, the NN shown in FIG. 4 includes a pyramidal feature network (Feature Pyramid Network (FPN)) in the feature extraction layer, and further includes a regression layer. In FIG. 4, the ResNet and FPN in the feature extraction layer calculate features related to an input image using a method called "Fully Convolutional Network (FCN)". The classification layer outputs the class of an object appearing in the image based on the features. The regression layer outputs the position of an object appearing in the image based on the features. For example, the regression layer outputs BB as the position.

[0027] For example, a car surveillance system can be constructed using an object detection task. More specifically, for example, the surveillance system acquires an image from a surveillance camera, inputs the acquired image into an object detection task, and acquires the class and position of a car appearing in the image from the object detection task. The surveillance system then displays the class and position of the car by superimposing it on the image from the surveillance camera. A user of the surveillance system can determine the car appearing in the surveillance camera image by using the displayed class and position of the car. The object detection task may also be used in combination with other recognition processes. For example, a license plate recognition system can be constructed by combining the following tasks: Object detection task: detecting the location and class of car license plates in an image. A region determination task to determine candidate regions for each character on the detected license plate. An image classification task in which images of determined candidate areas are used as input to classify the characters appearing in the images. The object detection task may detect the position and class of each character on a license plate included in an image.

[0028] Inference processing of image classification tasks and object detection tasks is required to have a high throughput. For example, image classification tasks and object detection tasks using video from a surveillance camera need to operate at a relatively high frame rate to prevent oversight. In particular, when targeting fast-moving objects such as cars, image classification tasks and object detection tasks need to operate at a high frame rate. For example, an object detection task for detecting cars in video from a surveillance camera should operate on video with a frame rate of 100 frames per second (fps) or more. Alternatively, a surveillance camera or a device around the surveillance camera, that is, an edge, may execute an image classification task or an object detection task. In such an edge environment, only limited computational resources may be available due to constraints such as installation space, cooling, and power. And, image classification tasks and object detection tasks are required to operate at a high throughput even in such an edge environment.

[0029] An AI device can execute inference processing with high throughput. However, since an AI device achieves high throughput by operating many computing units in parallel, it is necessary to operate many computing units at a high operating rate. As already explained, increasing the size of data to be inferred is an effective method for increasing parallelism. Therefore, similarly to the first embodiment, the information processing device 100 of the second embodiment improves the throughput of inference processing by using configuration data, which is data larger in size than the data, for inference on a target included in the data, as will be explained below.

[0030] [Overview of the second embodiment] In the following description, the information processing device 100 of the second embodiment uses an image, which is two-dimensional data, as an example of data. Then, the information processing device 100 executes an image classification task of classifying an image as an example of an inference task. More specifically, the information processing device 100 executes an image classification task in which an image containing characters, that is, a character image, is the classification target. For example, the information processing device 100 acquires an image of each character on a license plate (License Plate (LP)) of a vehicle, and infers the class of the character included in the image. That is, the character included in the image is the subject of inference. Also, inference of the class of the character is inference for the target.

[0031] However, the information processing device 100 does not directly infer the class of the character in the acquired image, but infers the class of the character using a constituent image composed of the acquired image. Specifically, the information processing device 100 generates a constituent image composed of a plurality of images arranged on a plane, and infers the class of the character in the generated constituent image. In other words, the information processing device 100 generates a constituent image that is an image larger in size than the original image by grouping together a plurality of images, at least some of which contain the character that is the subject of inference. Then, the information processing device 100 executes inference on the generated constituent image, and infers the class of the character included in the constituent image. In detail, the information processing device 100 infers the position and class of each character in a constituent image composed of a plurality of images containing characters.

[0032] Note that the image classification task is a task for inferring the class of an object contained in an image. However, the image classification task basically infers the class of an object in an image containing one classification target. Therefore, in the image classification task, the parallelism of the inference process is not necessarily high. On the other hand, the object detection task is likely to have higher parallelism than the image classification task because the object detection task infers the position and class of each object in an inference using an image containing multiple objects. Therefore, in the following description, the information processing device 100 uses an object detection task in the inference of the constituent images. Using such a configuration, when the information processing device 100 executes the inference process using an AI device, the information processing device 100 improves the utilization efficiency of the AI ​​device and improves the throughput of the inference process. However, these are not intended to limit the inference task used by the information processing device 100. The information processing device 100 may execute an appropriate inference task in response to the data containing the inference target and the generated constituent data, without being limited to the image classification task and the object detection task. The information processing device 100 may use the same type of task as the inference of both the image and the constituent images. Furthermore, the description of this embodiment is not intended to limit data that includes the processing target of the information processing device 100 to images.

[0033] [Configuration Description] FIG. 5 is a block diagram showing an example of the configuration of the information processing device 100 according to the second embodiment. The information processing device 100 includes an object inference unit 10, a data configuration unit 20, a configuration data inference unit 30, and a data inference unit 40. Furthermore, the information processing device 100 includes a data set storage unit 50, a data set generation unit 60, a model learning unit 70, a model storage unit 80, and a data acquisition unit 90. Note that the number and connection relationship of the components shown in FIG. 5 are exemplary. For example, the information processing device 100 may include a plurality of data acquisition units 90. Alternatively, the information processing device 100 may include a plurality of sets of the object inference unit 10, the data configuration unit 20, the configuration data inference unit 30, and the data inference unit 40, and each set may execute the operation described below in parallel to execute inference on objects included in a plurality of images in parallel.

[0034] The information processing device 100 may be configured using a computer device including a CPU, an AI device, a main memory, and a secondary storage device. In this case, the components of the information processing device 100 shown in FIG. 5 are realized using a combination of hardware and software such as a CPU. In this case, the information processing device 100 uses an AI device in all or part of the inference processing described below. The hardware configuration of the CPU and the like will be described in more detail later.

[0035] In the information processing device 100, the configuration for controlling the operation of each component is arbitrary. For example, the information processing device 100 may include a control unit (not shown) that controls each component. Alternatively, a predetermined component may control the operation of all or some of the other components. For example, the target inference unit 10 may control the operation of the data configuration unit 20, the configured data inference unit 30, and the data inference unit 40 to infer the class of characters in an image. Alternatively, each component may start operating when it acquires data from other components. Therefore, in the following description, a description of the control of the operation of the components will be omitted unless particularly necessary.

[0036] (1) Data Composition Section 20 The data construction unit 20 acquires an image including at least a part of the inference target from the data acquisition unit 90. Then, the data construction unit 20 uses a plurality of acquired images to generate a constituent image that is an image suitable for inference by the constituted data inference unit 30. The data construction unit 20 is an example of the data construction unit 21. In the following description, specifically, the data construction unit 20 combines a plurality of images acquired from the data acquisition unit 90 to generate a constituent image that is larger in size than the acquired image. Then, the data construction unit 20 generates a correspondence relationship between the image used to generate the constituent image and the position of the image in the constituent image. Alternatively, the data construction unit 20 may generate a correspondence relationship between the image used to generate the constituent image and the position of the area in the constituent image where the image is to be copied.

[0037] In the operations of the information processing device 100 described below, the operations related to the above two correspondence relationships are generally similar. Therefore, in the following description, in order to avoid complexity of the description, it is assumed that the "correspondence relationship between an image and its position in the constituent images" includes the "correspondence relationship between an image and the position of an area in the constituent images to which the image is to be copied." In other words, in the operations related to the following correspondence relationships, the information processing device 100 may use the "position of an area in the constituent images" instead of the "position of the image in the constituent images" in the operations related to the "correspondence relationship between an image and its position in the constituent images."

[0038] It is sufficient that at least a part of the image acquired by the data construction unit 20 includes the subject of inference. In other words, a part of the image acquired by the data construction unit 20 does not need to include the subject of inference. For example, if the subject of inference is a character included in the image, a part of the image acquired by the data construction unit 20 may be an image that does not include the character that is the subject of inference, or an image in which a part of the character is missing.

[0039] Next, an example of the generation of the constituent images in the data construction unit 20 will be described. However, the generation of the constituent images in the data construction unit 20 is not limited to the following. The data construction unit 20 divides the region of the constituent images into a plurality of grid-like regions based on at least some of the following parameters: Example parameters: (a) The size of the component images, i.e. their vertical and horizontal dimensions or width and height dimensions. (b) The number of horizontal and vertical grid lines. (c) The interval between the regions, i.e., the length of the gap between the regions. Note that the length of the gap may be different in the vertical and horizontal directions. (d) Pixel values ​​to fill the gaps between the regions. Pixel values ​​can be black, white, gray, or random.

[0040] For example, if the vertical and horizontal numbers of regions in a constituent image are used as parameters, the data construction unit 20 divides the constituent image into regions in the vertical and horizontal directions equal to the number acquired as parameters. For example, if the vertical number is 3 and the horizontal number is 4, the data construction unit 20 divides the constituent image into 12 (=3×4) regions. The data construction unit 20 acquires these parameters from an operator, etc.

[0041] Then, the data construction unit 20 copies each of the acquired images to each of the divided regions to generate the constituent images. If the size of the region in the constituent images does not match the size of the image, the data construction unit 20 may enlarge or reduce the image and copy it to the region. In this case, the data construction unit 20 may deform the image so as to maintain the aspect ratio of the image, or may enlarge or reduce the image in the horizontal and vertical directions independently. Alternatively, the data construction unit 20 may store a plurality of parameter sets and generate the constituent images from the image by switching the parameter sets according to the characteristics of the image. For example, the data construction unit 20 may select parameters to be used in generating the constituent images so that the amount of enlargement and reduction of the image is minimized. However, the data construction unit 20 may copy the image to the region without deforming it.

[0042] FIG. 6 is a diagram showing an example of a constituent image generated by the data composition unit 20. The upper part of FIG. 6 shows the images acquired by the data composition unit 20, that is, the group of images to be inferred. The lower part of FIG. 6 shows the constituent image generated by the data composition unit 20. In the constituent image at the bottom of FIG. 6, the region in which the image is duplicated and separated by the dotted line is the region in which the image is duplicated. The remaining part is the gap region between the regions. In FIG. 6, the data composition unit 20 divides the region of the constituent image into four regions in the horizontal direction and three regions in the vertical direction as a grid of the constituent image. Furthermore, the data composition unit 20 uses "white" as the pixel value of the gap to fill the gap between the images. Then, the data composition unit 20 duplicates the image from the left image of the image to be inferred to the right region in each of the upper region to the lower region of the constituent image. However, the position and order in which the images are duplicated in the constituent image are not limited to this. It should be noted that the dotted lines in the constituent images at the bottom of FIG. 6 are provided for the purpose of explanation and are not included in the constituent images that are actually generated.

[0043] The data constructing unit 20 may generate a constituent image by combining images based on channels, rather than duplicating the images in the planar direction. Note that a channel in an image is a mechanism for treating various information in an image as a grayscale image. Alternatively, the data constructing unit 20 may generate a constituent image by combining the planar direction and the channel direction.

[0044] (2) Configuration data inference unit 30 The constituted data inference unit 30 infers the position and class of a character in the constituted image using the constituted image generated by the data constructing unit 20 as an input. The constituted data inference unit 30 is an example of the constituted data inference unit 31. In the following description, specifically, the constituted data inference unit 30 infers the position and class of a character in the constituted image using an object detection task. The constituted data inference unit 30 uses a trained model stored in the model storage unit 80 in the inference. Hereinafter, the trained model used by the constituted data inference unit 30 may be referred to as a "constituted data inference model". The constituted data inference unit 30 may perform the inference after changing at least one of the shape and size of the constituted image.

[0045] (3) Data Inference Section 40 The data inference unit 40 infers the class of characters in an image by using at least a part of the image acquired by the data acquisition unit 90 as an input. The data inference unit 40 is an example of the data inference unit 41. Specifically, the data inference unit 40 infers the class of characters included in an image by using an image classification task. The data inference unit 40 uses a trained model stored in the model storage unit 80 in the inference. Hereinafter, the trained model used by the data inference unit 40 may be referred to as a "model for data inference". The data inference unit 40 performs inference on an image specified. For example, the data inference unit 40 may perform inference on an image specified by the target inference unit 10. The target inference unit 10 may instruct the data acquisition unit 90 on an image to be output to the data inference unit 40. In this case, the data inference unit 40 may perform inference on all images acquired from the data acquisition unit 90. The data inference unit 40 may perform inference after changing at least one of the shape and size of the image.

[0046] In the description of the second embodiment, the inference made by the information processing device 100 is an inference of a class of characters included in an image. That is, in the description of the second embodiment, the inference made by the data inference unit 40 is the same as the inference made by the information processing device 100. However, the inference made by the data inference unit 40 may be different from the inference made by the information processing device 100 as long as it outputs an inference result similar to that of the inference made by the information processing device 100.

[0047] The difference between the inference in the constituted data inference unit 30 and the data inference unit 40 will be described. The constituted image is an image obtained by combining images acquired by the data acquisition unit 90, and is an image larger in size than the image acquired by the data acquisition unit 90. In other words, the constituted image that is the subject of inference in the constituted data inference unit 30 is an image larger in size than the image that is the subject of inference in the data inference unit 40. Furthermore, the object detection task used by the constituted data inference unit 30 is an inference task that is more likely to execute an inference process with higher parallelism than the image classification task used by the data inference unit 40. Therefore, the constituted data inference unit 30 can use multiple arithmetic units in an AI device more efficiently than the data inference unit 40.

[0048] In this way, the trained model used by the configured data inference unit 30 is a model for images larger in size than the trained model used by the data inference unit 40. Therefore, the trained model used by the configured data inference unit 30 is a model larger in scale than the trained model used by the data inference unit 40. Note that the trained model used by the configured data inference unit 30 may be a model having at least one of the same model structure and network structure as the trained model used by the data inference unit 40. However, the trained model used by the configured data inference unit 30 may be a model having both a different model structure and a different network structure from the trained model used by the data inference unit 40. (4) Object Inference Section 10 The target inference unit 10 performs inference on the target in the image acquired by the data acquisition unit 90. The target inference unit 10 is an example of the target inference unit 11. Specifically, the target inference unit 10 first determines, for each image included in the constituent image, whether the inference on the target included in the image constituting the constituent image in the constituted data inference unit 30 has been successful or unsuccessful. In the case of an image in which the inference on the target included in the image constituting the constituent image in the constituted data inference unit 30 has been successful, the target inference unit 10 infers the class of characters of that image based on the inference result in the constituted data inference unit 30 and the constituent image.

[0049] Specifically, the object inference unit 10 infers the class of characters contained in the image using the following information. (a) the position and class of characters as a result of inference by the configuration data inference unit 30; (b) The correspondence relationship generated by data construction unit 20 between the images used to generate the constituent images and the positions of those images in the constituent images. For example, the target inference unit 10 extracts an image position that includes the position of the character inferred by the constituted data inference unit 30 from among the image positions in the constituted images. Then, the target inference unit 10 identifies an image that corresponds to the position of the extracted image, i.e., the image used to generate the constituted images, based on the correspondence between the image used to generate the constituted images and the position of that image in the constituted images. Then, the target inference unit 10 infers that the class of the character inferred by the constituted data inference unit 30 is the class of the character in the identified image. Then, the target inference unit 10 outputs the inference result.

[0050] In the case of an image in which the inference of an object included in an image constituting a constructed image in the constructed data inference unit 30 has failed, the object inference unit 10 instructs the data inference unit 40 to infer an object included in the image, and acquires an inference result from the data inference unit 40. Then, the object inference unit 10 infers that the class of characters, which is the inference result in the data inference unit 40, is the class of characters in the image. Then, the object inference unit 10 outputs the inference result. Note that the image acquired by the data construction unit 20 may be an image that does not include characters to be inferred. Therefore, when the inference in the data inference unit 40 has failed, the object inference unit 10 may output an inference result of an image that does not include characters as the inference result for the object included in the image. Alternatively, the object inference unit 10 may output, as the inference result, "that inference of the object included in the image has failed."

[0051] Next, the determination of whether the inference of the constituted data inference unit 30 in the target inference unit 10 is successful or unsuccessful will be described. In the description of this embodiment, the target inference unit 10 uses the position of the character inferred in the constituted image to determine whether the inference of the constituted data inference unit 30 is successful or unsuccessful. Details are as follows. The constituted data inference unit 30 infers the position of the BB corresponding to the character as the position of the character. Then, the target inference unit 10 determines whether the inference is successful or unsuccessful based on the position of the BB and the position of the image copied to the constituted image generated by the data constructing unit 20. For example, the target inference unit 10 determines an image including a BB whose position of an image constituting the constituted image is inferred as an image for which inference has been successful. Note that the image including the BB includes an image that matches the BB, that is, an image of the same shape as the BB, in addition to an image that includes the BB inside. Alternatively, the target inference unit 10 may determine an image in which the deviation between the position of the image constituting the constituted image and the position of the BB is within a predetermined range as an image for which inference has been successful, taking into account an error in inference. On the other hand, the target inference unit 10 determines that an image in which a BB is not inferred at the position of an image constituting a constituent image, and an image in which a part of the BB is not included at the position of an image constituting a constituent image, is an image in which inference has failed.

[0052] However, the judgment in the target inference unit 10 is not limited to the relationship between the position of the image in the above-mentioned constituted image and the position of the BB. For example, the target inference unit 10 may use the certainty of the inference in the constituted data inference unit 30. Alternatively, the target inference unit 10 may use the IoU (Intersection over Union) between the image in the constituted image and the BB to judge whether the inference of the constituted data inference unit 30 has succeeded or failed. The IoU is one of the evaluation indices in object detection, and is an index that indicates how much two regions overlap. Specifically, the IoU is the ratio of the intersection of the two regions to the union of the two regions. For example, the target inference unit 10 may judge an image in which the IoU between the region of the image in the constituted image and the region of the BB is higher than a predetermined threshold value as an image in which the inference has been successful. However, the target inference unit 10 may use a judgment criterion different from the above to judge whether the inference of the constituted data inference unit 30 has succeeded or failed.

[0053] (5) Dataset Generation Unit The dataset generating unit 60 generates a learning dataset used for machine learning to generate a trained model using a predetermined dataset. In detail, the dataset generating unit 60 generates a learning dataset for machine learning to generate a configured data inference model to be used for inference in the configured data inference unit 30. Hereinafter, the learning dataset for generating the configured data inference model may be referred to as a "configured data learning dataset." Furthermore, the dataset generating unit 60 generates a learning dataset for machine learning to generate a data inference model to be used for inference in the data inference unit 40. Hereinafter, the learning dataset for generating a data inference model may be referred to as a "data learning dataset."

[0054] In the following description, the dataset used by the dataset generating unit 60 to generate the learning dataset may be referred to as an "original dataset". For example, the dataset generating unit 60 may generate each learning dataset using an original dataset acquired from a user. However, the source of the original dataset is not limited to the above. For example, the dataset generating unit 60 may generate each learning dataset using an original dataset stored in advance in the dataset storage unit 50. The dataset generating unit 60 may generate both the constituent data learning dataset and the data learning dataset using the same original dataset. Alternatively, the dataset generating unit 60 may generate the constituent data learning dataset and the data learning dataset using different original datasets.

[0055] When generating a constituent data learning dataset, the dataset generating unit 60 may generate constituent images from images included in the original dataset. In this case, the dataset generating unit 60 may generate constituent images from images included in the original dataset using an operation similar to that of generating constituent images in the data constructing unit 20. The dataset generating unit 60 may generate correct answer data for the generated constituent images using information on the position of the image in the constituent images. For example, when the constituent data inference unit 30 uses an object detection task, the dataset generating unit 60 may use a predetermined object detection task to create the following data as correct answer data for the constituent images for each image used in generating the constituent images. (a) BB: The position of the characters in the image in the constructed image generated from the original dataset. (b) Class: The class of characters contained in the image in the constructed image generated from the original dataset. The data set generation unit 60 may estimate the class of the images included in the original data set as the class of the correct answer data by using a predetermined image classification task.

[0056] The method of generating correct answer data is sometimes called annotation. Annotation generally refers to adding information related to metadata, etc., as annotations to certain data. In the field of machine learning, however, annotation refers to creating learning data, correct answer data, labels, etc. for training a machine learning model. For example, the role of annotation is to give meaning or link data such as images and combine them with each other. The learning dataset is sometimes called "teacher data."

[0057] (6) Dataset storage section The dataset storage unit 50 stores the learning dataset generated by the dataset generation unit 60. For example, when the configuration data inference unit 30 uses a trained model for an object detection task, the dataset storage unit 50 stores a configuration data learning dataset including an image to be subjected to object detection and correct answer data such as BB and class in the image. Alternatively, when the data inference unit 40 uses a trained model for an image classification task, the dataset storage unit 50 stores a data learning dataset including an image to be classified and correct answer data such as class.

[0058] (7) Model learning section The model learning unit 70 generates a trained model used by the configured data inference unit 30 and the data inference unit 40. Specifically, the model learning unit 70 generates a configured data inference model used by the configured data inference unit 30 by machine learning using the configured data learning dataset stored in the dataset storage unit 50. Furthermore, the model learning unit 70 generates a data inference model used by the data inference unit 40 by machine learning using the data learning dataset stored in the dataset storage unit 50. Then, the model learning unit 70 stores the generated trained model in the model storage unit 80.

[0059] (8) Model storage section The model storage unit 80 stores the trained model generated by the model learning unit 70 and used for inference by the configuration data inference unit 30, i.e., the trained model for configuration data inference. Furthermore, the model storage unit 80 stores the trained model generated by the model learning unit 70 and used for inference by the data inference unit 40, i.e., the model for data inference.

[0060] (9) Data Acquisition Section The data acquisition unit 90 acquires an image including an object of inference of the information processing device 100. The data acquisition unit 90 acquires an image from, for example, a surveillance camera. Alternatively, the data acquisition unit 90 may acquire an image from a storage device (not shown) that stores an image including an object of inference. The data acquisition unit 90 may be included in a device that acquires images, such as a surveillance camera.

[0061] [Explanation of operation] The operation of generating a trained model and the operation of executing inference on a target in the information processing device 100 according to the second embodiment will be described with reference to the drawings.

[0062] (1) Generating a trained model FIG. 7 is a flow diagram showing an example of an operation of generating a trained model in the information processing device 100 according to the second embodiment. The information processing device 100 starts the operation when a predetermined condition is met. For example, the information processing device 100 starts the operation of generating a trained model when an instruction from an operator of the information processing device 100 is received. In this case, at the start of the operation, the information processing device 100 may acquire parameters necessary for generating the trained model from the operator. The parameters are, for example, a specification of a learning dataset to be used for generating the model, or parameters related to machine learning. However, the parameters are not limited to these and may be other information. The information processing device 100 may acquire information different from the parameters. For example, the information processing device 100 may acquire an original dataset, correct answer data, or a learning dataset from the operator. In this case, the information processing device 100 may store the acquired original dataset, correct answer data, or learning dataset in the dataset storage unit 50. Note that the correct answer data may be included in the learning dataset.

[0063] The dataset generation unit 60 acquires an original dataset from the dataset storage unit 50, and generates a constituted data learning dataset using the acquired original dataset (step S101). The dataset generation unit 60 may acquire correct answer data of the constituted data learning dataset from the dataset storage unit 50. Then, the dataset generation unit 60 stores the generated constituted data learning dataset in the dataset storage unit 50. Furthermore, the dataset generation unit 60 acquires an original dataset from the dataset storage unit 50, and generates a data learning dataset using the acquired original dataset (step S102). In this case, too, the dataset generation unit 60 may acquire correct answer data of the data learning dataset from the dataset storage unit 50. Then, the dataset generation unit 60 stores the generated data learning dataset in the dataset storage unit 50. The dataset generation unit 60 may execute the operations of steps S101 and S102 in a reverse order, or may execute at least some of the operations in parallel.

[0064] The model learning unit 70 generates a configured data inference model by machine learning using the configured data learning data set (step S103). The configured data inference model is a trained model used by the configured data inference unit 30. The model learning unit 70 stores the generated configured data inference model in the model storage unit 80. Furthermore, the model learning unit 70 generates a data inference model by machine learning using the data learning data set (step S104). The data inference model is a trained model used by the data inference unit 40. The model learning unit 70 stores the generated data inference model in the model storage unit 80. The model learning unit 70 may execute the operations of steps S103 and S104 in a reverse order, or may execute at least some of the operations in parallel. After the operation of generating the model is completed, the information processing device 100 may notify the operator of the operation result.

[0065] (B) The act of performing inferences about an object FIG. 8 is a flow diagram showing an example of an operation of executing inference on a target in the information processing device 100 according to the second embodiment. The information processing device 100 starts the inference operation when a predetermined condition is met. For example, the information processing device 100 starts the inference operation when an instruction from an operator is met. Alternatively, the information processing device 100 may automatically start the inference operation after the device is started. At the start of the inference operation, the information processing device 100 may acquire parameters related to the inference operation from the operator. Note that the parameters are, for example, designation of a trained model to be used, but are not limited thereto. However, the information processing device 100 may use parameters acquired in advance. At the start of the inference operation, if the trained model to be used has not been saved, the information processing device 100 may execute the above-mentioned "(A) Operation of generating a trained model" to generate a trained model to be used, and then start the inference operation.

[0066] The configuration data inference unit 30 and the data inference unit 40 acquire a trained model to be used for inference from the model storage unit 80 (step S110). Then, the information processing device 100 repeats the operation of loop A until a predetermined end condition is satisfied (step S111). An example of the end condition is an end instruction from an operator. However, the information processing device 100 may end the operation of loop A based on other end conditions such as an operation time or the number of images.

[0067] In loop A, first, the data construction unit 20 repeats the operation of loop B until a predetermined image condition is satisfied (step S112). One example of an image condition is the number of images. That is, the data construction unit 20 repeats the operation of loop B until the predetermined number of images is reached. However, the data construction unit 20 may end loop B based on other image conditions, such as the total area of ​​the acquired images or the total amount of data of the acquired images. In loop B, the data construction unit 20 acquires images to be used for generating the constituent images from the data acquisition unit 90 (step S113). In the following description, the images acquired in one operation of loop B are collectively referred to as a "group of images to be constructed."

[0068] When the image conditions are satisfied, the data construction unit 20 generates a constructed image from the acquired group of constructed target images (step S114). The data construction unit 20 generates the constructed image, for example, by duplicating the image of the group of constructed target images in an area of ​​the constructed image arranged in a grid pattern. Next, the constructed data inference unit 30 executes inference using the generated constructed image as an input (step S115). For example, the constructed data inference unit 30 infers a pair of the class of the character included in the constructed image, BB, and certainty using a trained model of the object detection task.

[0069] Then, the target inference unit 10 repeats the operation of loop C for all images copied in the constituent image, that is, all images included in the group of constituent target images (step S116). In loop C, the target inference unit 10 judges whether the inference by the constituted data inference unit 30 has succeeded or failed for each image of the group of constituent target images in the constituent image (step S117). For example, the target inference unit 10 compares the position of the image in the constituent image with the position of BB in the inference result of the constituted data inference unit 30 to judge whether the inference has succeeded or failed.

[0070] In the case of an image in which the inference in the constituent image is successful (Yes in step S117), the target inference unit 10 adopts the inference result for the object included in the constituent image as the inference result for the object included in the image (step S118). For example, when the information processing device 100 executes an image classification task and the constituted data inference unit 30 executes an object detection task, the target inference unit 10 adopts the class and certainty included in the inference result of the constituted data inference unit 30 as the inference result for the object included in the image. In the case of an image in which the inference for the object included in the constituent image is unsuccessful (No in step S117), the target inference unit 10 instructs the data inference unit 40 to infer the object included in the image. The data inference unit 40 executes inference for the object included in the image. Then, the target inference unit 10 adopts the inference result for the object included in the image as the inference result for the object included in the image (step S119). Using such an operation, the information processing device 100 executes inference for the object included in the image acquired by the data acquisition unit 90.

[0071] [Throughput explanation] Next, an example of comparison between the throughput in the information processing device 100 of the second embodiment and the throughput when only the image classification task is used will be described. Hereinafter, the case when only the image classification task is used will be referred to as the "comparison technique." The following explanation is based on the following assumptions. Note that the load of processes other than the inference process is smaller than the inference process, and will be ignored in the following explanation. The unit of throughput is fps. Processing volume The size of the image to be inferred by the comparison technique and data inference unit 40 is 64×64 pixels. The size of the constituted image to be inferred by the constituted data inference unit 30 is 512×512 pixels. In addition, the increase in the processing amount of the object detection task relative to the image classification task when the feature extraction layer is the same is assumed to be 1.2 times. In this case, the processing amount of the constituted data inference unit 30 is (512×512) / (64×64)×1.2=76.8 times that of the comparison technique and data inference unit 40. - Efficiency of AI device usage The utilization efficiency of the AI ​​device when an image is used is assumed to be 10%. Also, the utilization efficiency of the AI ​​device when a constituted image is assumed to be 70%. In other words, the utilization efficiency of the AI ​​device in constituted data inference unit 30 is 70 / 10=7 times that of the comparison technology. Configuration image The number of regions in which images are replicated in the constituent images is 7 x 7. In other words, the constituent images are images in which 49 images are replicated. ·variable The variables included in the formulas used in the explanation are as follows: CPS comp : Comparison technique and throughput of the AI ​​device in image units in the inference of the data inference unit 40 [unit: character per second (cps)], FPS max : Maximum throughput of the AI ​​device per constituent image in the inference of the constituent data inference unit 30 [unit: fps], CPS max : Maximum throughput of the AI ​​device per image in the inference of the configuration data inference unit 30 [unit: cps], CPS 10%failure :Throughput of the AI ​​device per image when the inference of the configuration data inference unit 30 fails 10% of the time [unit: cps].

[0072] In this case, FPS max and CPS max teeth, FPS max =(CPS comp / (((512×512) / (64×64))×1.2))×(70 / 10) =(CPS comp / 76.8)×7 ≒0.091 × CPS comp CPS max =FPS max ×49 =0.091 x CPS comp ×49≒4.46×CPS comp In other words, the throughput CPS when the constituted data inference unit 30 succeeds in inferring the objects included in all constituted images is max The throughput of the compared technologies is CPS comp This is about 4.46 times the CPS max is the maximum throughput of the information processing device 100.

[0073] Throughput CPS when 10% of the inferences made by the configuration data inference unit 30 fail 10%failure First, the processing time of the configuration data inference unit 30 and the processing time of the data inference unit 40 are respectively as follows: Processing time of the configuration data inference unit 30: N / (CPS max )≒N / (4.46×CPS comp ), Processing time of data inference unit 40: 0.1×N / CPS comp Here, N is the number of images to be processed. From the total processing time of the configuration data inference unit 30 and the processing time of the data inference unit 40 and the number of images (N), the throughput CPS in this case is calculated as follows: 10%failure teeth, CPS 10%failure =N / (N / (4.46×CPS comp ) + 0.1 × N / CPS comp ) =CPS comp / (1 / 4.46+0.1) ≒3.08 × CPS comp In other words, even if 10% of the inferences made by the configuration data inference unit 30 fail, the throughput of the information processing device 100 is about 3.08 times the throughput of the comparison technique.

[0074] In this way, the information processing device 100 improves the throughput of the inference process. The reason is as follows. The information processing device 100 includes an object inference unit 10, a data configuration unit 20, a configuration data inference unit 30, and a data inference unit 40. The data configuration unit 20 generates configured data configured using a plurality of data, at least a part of which includes an inference target. The configuration data inference unit 30 executes inference for an object included in the configured data. The data inference unit 40 executes inference for an object included in at least a part of the data. The object inference unit 10 judges whether inference for an object included in the configured data has succeeded or failed for each piece of data constituting the configured data. Then, the object inference unit 10 executes inference for an object included in the configured data, based on the inference result for the object included in the configured data and the configured data, as an inference for an object included in data, in which inference for an object included in the configured data has succeeded. Then, the object inference unit 10 executes inference for an object included in data, in which inference for an object included in the configured data has failed, based on the inference result for the object included in the data.

[0075] The information processing device 100 performs inference using configuration data, which is data larger in scale than the data, and improves the throughput of the inference process. In particular, when using hardware that achieves high throughput by operating a large number of computing units in parallel, such as an AI device, the information processing device 100 can improve the throughput of the inference process. For example, in the above description of the throughput, the maximum throughput of the information processing device 100 is about 4.46 times the throughput of the comparison technology.

[0076] Furthermore, for data for which inference on the target has failed, the target inference unit 10 executes inference on the target using the inference result of the data inference unit 40. The inference in the data inference unit 40 is an inference with a lower degree of parallelism than the inference in the configuration data inference unit 30. However, the data inference unit 40 performs inference on data smaller in scale than the configuration data, using a model smaller in scale than the configuration data inference unit 30. Therefore, the inference in the data inference unit 40 is an inference with a lower load than the inference in the configuration data inference unit 30. Also, when the probability of failure in the configuration data inference unit 30 is low, the amount of data inferred by the data inference unit 40 is significantly smaller than the amount of data to be inferred. Therefore, in this case, even if the information processing device 100 uses the data inference unit 40, the impact on the performance of the entire inference can be kept small.

[0077] Moreover, the data inferred by the data inference unit 40 is data that the configured data inference unit 30 failed to infer. Therefore, when the configured data inference unit 30 executes inference again on the target included in that data, it is assumed that there is a high possibility of failure. In contrast, it is assumed that inference using the data inference unit 40 has a lower possibility of inference failure than inference using the configured data inference unit 30 again. As a result, by performing inference using the data inference unit 40, the information processing device 100 can reduce the performance degradation of the entire process due to the failure of re-inference in the configured data inference unit 30.

[0078] In this way, the information processing device 100 uses the data inference unit 40 to improve the reduction in throughput even when there is data for which inference is not successful in the configured data inference unit 30. For example, in the above described example, even if the inferences of the configured data inference unit 30 fail 10% of the time, the throughput of the information processing device 100 is about 3.08 times that of the comparative technique.

[0079] Furthermore, the information processing device 100 may include a data set generating unit 60, a data set saving unit 50, a model learning unit 70, a model saving unit 80, and a data acquiring unit 90. The data set generating unit 60 generates a configured data learning data set for generating a configured data inference model by using a predetermined data set. Furthermore, the data set generating unit 60 generates a data learning data set for generating a data inference model by using a predetermined data set. The data set saving unit 50 saves the configured data learning data set and the data learning data set. The model learning unit 70 generates a configured data inference model by machine learning using the configured data learning data set. Furthermore, the model learning unit 70 generates a data inference model by machine learning using the data learning data set. The model saving unit 80 saves the configured data inference model and the data inference model. The data acquiring unit 90 acquires data. Then, the data configuring unit 20 generates configured data configured from the data acquired from the data acquiring unit 90. The configured data inference unit 30 executes inference on a target included in the configured data by using the configured data inference model stored in the model saving unit 80. The data inference unit 40 acquires data instructed by the object inference unit 10 from the data acquisition unit 90, and infers an object included in the acquired data using a data inference model stored in the model storage unit 80. Based on this configuration, the information processing device 100 generates a model to be used for inference, and executes inference on an object included in the data acquired by the data acquisition unit 90 using the generated model.

[0080] The information processing device 100 may use two-dimensional data such as an image as the data. For example, when the data is an image, the data construction unit 20 may generate a constructed image in which a plurality of images are arranged on a two-dimensional plane. In this case, the constructed data inference unit 30 may execute an object detection task as an inference for an object included in the constructed image. Furthermore, in this case, the data inference unit 40 may execute an image classification task as an inference for an object included in the image. The constructed image includes a plurality of images. Furthermore, the object detection task is more likely to execute a process with a higher degree of parallelism than the image classification task as an inference process for an object included in a plurality of images. Therefore, the information processing device 100 improves the throughput of the inference process.

[0081] [Variations] The dataset storage unit 50 may store, as at least a portion of the training dataset, a training dataset provided in advance by a user or the like. Alternatively, the information processing device 100 may acquire, as at least a portion of the training dataset, a training dataset generated in another device (not shown) and store the acquired training dataset in the dataset storage unit 50. When the information processing device 100 acquires all of the training datasets and stores them in the dataset storage unit 50, the information processing device 100 may not include the dataset generation unit 60.

[0082] The model learning unit 70 may acquire at least a part of the learning dataset from a device not shown. When the model learning unit 70 acquires all of the learning datasets from a device not shown, the information processing device 100 may not include the dataset storage unit 50 and the dataset generation unit 60.

[0083] The model storage unit 80 may store a trained model acquired from a user. Alternatively, the information processing device 100 may acquire a trained model generated in a device not shown and store it in the model storage unit 80. When the information processing device 100 acquires all trained models and stores them in the model storage unit 80, the information processing device 100 does not need to include the dataset storage unit 50, the dataset generation unit 60, and the model training unit 70.

[0084] At least one of the configuration data inference unit 30 and the data inference unit 40 may obtain the trained model to be used from another device. When both the configuration data inference unit 30 and the data inference unit 40 obtain the trained model from an external device not shown, the information processing device 100 does not need to include the dataset storage unit 50, the dataset generation unit 60, the model learning unit 70, and the model storage unit 80.

[0085] The information processing device 100 may include a plurality of pairs of the data constructing unit 20 and the configured data inference unit 30 as pairs of the data constructing unit 20 and the configured data inference unit 30. For example, the information processing device 100 may include two or more pairs of the data constructing unit 20 and the configured data inference unit 30. FIG. 9 is a block diagram showing the configuration of an information processing device 103 which is an example of a variation of the information processing device 100 according to the second embodiment. The information processing device 103 includes a pair of a data constructing unit 23 and a configured data inference unit 33 in addition to the pair of the data constructing unit 20 and the configured data inference unit 30 which are components of the information processing device 100.

[0086] In the following description, when it is necessary to distinguish between the set of the data constructor 20 and the configuration data inference unit 30 and the set of the data constructor 23 and the configuration data inference unit 33, they may be called as follows. The set of the data composing unit 20 and the configured data inference unit 30 is a first set, the data composing unit 20 is a first data composing unit, the configured data inference unit 30 is a first configured data inference unit, and the configured data generated by the data composing unit 20 is the first configured data. The set of the data composing unit 23 and the configured data inference unit 33 is a second set, the data composing unit 23 is a second data composing unit, the configured data inference unit 33 is a second configured data inference unit, and the configured data generated by the data composing unit 23 is the second configured data.

[0087] The data configuration unit 23 generates second constituted data including a plurality of data using a plurality of data, similarly to the data configuration unit 21. For example, when the data is an image, the data configuration unit 23 generates the second constituted image by combining a plurality of images. However, the data configuration unit 23 generates the second constituted data using data for which the inference for the object included in the constituted data in the constituted data inference unit 30 has failed as data for generating the second constituted data. The constituted data inference unit 33 executes inference for the object included in the second constituted data generated by the data configuration unit 23. The object inference unit 10 judges whether the inference for the object included in the second constituted data has succeeded or failed for each piece of data constituting the second constituted data. Then, the object inference unit 10 executes inference for the object based on the inference result for the object included in the second constituted data and the second constituted data as an inference for the object included in the data for which the inference for the object included in the second constituted data has succeeded. Furthermore, the object inference unit 10 instructs the data inference unit 40 to infer the object included in the data for which the inference for the object included in the second constituted data has failed. Then, the object inference unit 10 performs inference on the object contained in the data based on the inference result for the object contained in the data, as an inference for the object contained in the data for which the inference for the object contained in the second constituted data has failed.

[0088] The second constituted data is composed of data that the constituted data inference unit 30 failed to infer. Therefore, when the constituted data inference unit 33 executes inference on a target included in the second constituted data using the same model as the constituted data inference unit 30, there is a high possibility that the inference will fail. Therefore, for example, it is desirable that the model used by the constituted data inference unit 33 is different from the model used by the constituted data inference unit 30. Alternatively, it is desirable that the second constituted data generated by the data constituted unit 23 is data in a format different from that of the constituted data generated by the data constituted unit 20. For example, in the case of a data image, the data constituted unit 23 may generate the second constituted data with a wider interval between images than the constituted data generated by the data constituted unit 20, or may generate the second constituted data after increasing the size of the image to be copied.

[0089] When the information processing device 100 includes a plurality of sets of the data configuration unit 20 and the configuration data inference unit 30, the information processing device 100 may use the plurality of sets in parallel, instead of using them in series as described above. For example, the target inference unit 10 may distribute the data acquired by the data acquisition unit 90 to one of the plurality of sets. For example, the target inference unit 10 may distribute the data to each set according to a round robin method. Alternatively, for example, when the inference method or model in the configuration data inference unit 30 differs in each set, the information processing device 100 may distribute the data to a set suitable for inference on the target included in the data, based on the attributes of the data, such as size. Note that the data acquisition unit 90 may generate metadata related to the acquired data, such as the attributes of the data. In this case, the information processing device 100 may use the generated metadata to distribute the data.

[0090] When a plurality of sets of the data constructing unit 20 and the constituted data inference unit 30 are used in parallel, the information processing device 100 may include a data inference unit 40 corresponding to each set. However, when the probability of failure of inference in the constituted data inference unit 30 is low, the information processing device 100 may include one data inference unit 40, and the data inference unit 40 may execute inference for objects included in images for which the constituted data inference units 30 of all sets have failed. Alternatively, the information processing device 100 may include a number of data inference units 40 fewer than the number of sets, and assign and cause the plurality of data inference units 40 to execute inference for objects included in images for which the constituted data inference unit 30 has failed.

[0091] When multiple sets of the data configuration unit 20 and the configured data inference unit 30 are used in parallel, the information processing device 100 may include a target inference unit 10 corresponding to each set. However, since the inference process of the target inference unit 10 is an inference process using the inference results of the configured data inference unit 30 and the data inference unit 40, the load is lower than that of the inference process of the configured data inference unit 30 and the data inference unit 40. Therefore, the information processing device 100 may use one target inference unit 10 to execute inference based on the inference results of multiple sets of the configured data inference unit 30 and the data inference unit 40. Alternatively, the information processing device 100 may include a target inference unit 10 for each predetermined number of sets.

[0092] Although the degree of improvement in throughput is reduced, the information processing device 100 may include one pair of the data configuration unit 20 and the configuration data inference unit 30, and may operate again using the data for which inference has failed in the configuration data inference unit 30 as the input for the data configuration unit 20. In particular, when the probability of failure of inference in the configuration data inference unit 30 is low, the information processing device 100 may repeat the operation of the data configuration unit 20 and the configuration data inference unit 30 again. That is, the information processing device 100 may repeat the operation of the data configuration unit 20 and the configuration data inference unit 30 up to a predetermined number of times for data for which inference has failed in the configuration data inference unit 30. However, when inferring configured data including failed data using the same model, inference for an object included in the data is likely to fail. Therefore, when repeating the operation for the same data, it is desirable for the information processing device 100 to change the operation of at least one of the data configuration unit 20 and the configuration data inference unit 30. In this way, the information processing device 100 may proceed with inference while changing the operation of at least one of the data configuration unit 20 and the configuration data inference unit 30.

[0093] In the above description, an example has been described in which the information processing device 100 executes an image classification task as an inference process, the constituted data inference unit 30 executes an object detection task, and the data inference unit 40 executes an image classification task. However, the second embodiment is not limited to this. The information processing device 100 may execute an inference task different from the image classification task. Alternatively, the constituted data inference unit 30 may execute an inference task different from the object detection task. Alternatively, the data inference unit 40 may execute an inference task different from the image classification task.

[0094] For example, the information processing device 100 may execute, as an inference, a re-identification task for calculating a numerical value that can be used to distinguish or identify multiple objects of the same type or class. The re-identification task is one of the object detection tasks, and is a task for distinguishing and re-identifying multiple objects that belong to the same category. In the case of the re-identification task, the information processing device 100 may omit the inference in the data inference unit 40. Note that, for example, when the model of the re-identification task is composed of a feature extraction layer and a re-identification layer, the trained model used by the configuration data inference unit 30 is configured by replacing the classification layer and regression layer in the model of the object detection task in FIG. 4 with a re-identification layer.

[0095] In the description of the second embodiment so far, the data construction unit 20 of the information processing device 100 divides the region of the constituent image into a plurality of partial regions in a grid shape, and copies the image to each region, as an example. However, the operation of the data construction unit 20 is not limited to this operation. For example, when the constituent data inference unit 30 executes an object detection task, the data construction unit 20 may generate the constituent image taking into consideration the anchor setting in the learned model of the object detection task used by the constituent data inference unit 30. More specifically, the data construction unit 20 may combine images based on the relationship between the anchor and the receptive field so as to improve the inference accuracy for the object included in the constituent image, which is the input of the object detection task. Note that the anchor is a method of performing object detection for each of a plurality of BBs having different aspect ratios determined in advance, and increasing the number of objects that can be detected simultaneously. Also, the receptive field is an area that can be processed in each layer of the model.

[0096] For example, when the data constructing unit 20 generates a constituted image by arranging images in a grid and the constituted data inference unit 30 executes an object detection task, the constituted data inference unit 30 may use anchor settings suitable for inferring objects included in the images that constitute the constituted image. For example, the learned model of the object detection task used by the constituted data inference unit 30 may be configured to use anchor settings suitable for inferring objects included in each image included in the constituted image.

[0097] <Third embodiment> The information processing device 102 according to the third embodiment changes at least one of the parameters related to the inference process and the inference operation according to the situation of the inference operation. The parameters related to the inference process are, for example, parameters that specify a trained model to be used for inference, but are not limited to the following. The inference operation to be changed is, for example, whether or not to use the configuration data inference unit 30, which will be described in more detail later. Hereinafter, the third embodiment will be described with reference to the drawings. In addition, in each drawing referred to in the description of the third embodiment, the same reference numerals are given to the configurations and operations as those of the second embodiment, and detailed description will be omitted as appropriate. In addition, the information processing device 102 may include a set of a plurality of data configuration units 20 and a configuration data inference unit 30, as in the second embodiment. However, in the following description, for convenience of description, the information processing device 102 includes only one configuration.

[0098] [Configuration Description] 10 is a block diagram showing an example of the configuration of an information processing device 102 according to the third embodiment. The information processing device 102 may be configured using a computer device including a CPU, as in the second embodiment. The information processing device 102 includes a target inference unit 12, a data configuration unit 20, a configured data inference unit 30, and a data inference unit 40. Furthermore, the information processing device 102 includes a data set storage unit 52, a data set generation unit 62, a model learning unit 72, a model storage unit 82, and a data acquisition unit 90.

[0099] In the information processing device 102, the configuration for controlling the operation of each component is arbitrary, as in the second embodiment. However, in the following explanation, for convenience of explanation, the target inference unit 12 changes parameters to control the operation of the data configuration unit 20 and the like. The data configuration unit 20, the configured data inference unit 30, and the data inference unit 40 are similar to the second embodiment except that they change their operations in response to the parameters changed by the target inference unit 12, so detailed explanations will be omitted. In addition, the data acquisition unit 90 is similar to the second embodiment, so detailed explanations will be omitted.

[0100] The target inference unit 12 operates in the same manner as the target inference unit 10 of the second embodiment, and executes inference on the target. Furthermore, the target inference unit 12 monitors the operating status of the information processing device 102, changes parameters related to inference according to the operating status, and switches operations such as inference. The operating status is, for example, the inference accuracy or inference speed of at least one of the configuration data inference unit 30 and the data inference unit 40. However, the operating status is not limited to these. Next, as an example of the operating status of the information processing device 102, an example in which the inference accuracy of the configuration data inference unit 30 falls below a predetermined threshold will be described. Note that the target inference unit 12 may use an accuracy used in general inference, such as recall, as the inference accuracy of the configuration data inference unit 30.

[0101] When the inference accuracy of the constituted data inference unit 30 falls below a predetermined threshold, the target inference unit 12 executes any one of the following operations or a combination of at least some of the operations. Note that when the inference accuracy of the constituted data inference unit 30 exceeds a predetermined threshold, the target inference unit 12 executes the opposite operation to the operation described below. (1) The target inference unit 12 temporarily stops the inference of the configured data inference unit 30. In other words, the target inference unit 12 does not use the inference result of the configured data inference unit 30, which has low accuracy, but uses the inference result of the data inference unit 40. Note that while the configured data inference unit 30 is stopped, the target inference unit 12 may stop the operation of the data configuration unit 20. The target inference unit 12 resumes the operation of the configured data inference unit 30 when a preset condition is satisfied, such as an instruction from an operator or the passage of a predetermined time. (2) The target inference unit 12 changes the value of the threshold used to determine whether the inference in the configuration data inference unit 30 has succeeded or failed. For example, when using certainty or IoU for the determination, the target inference unit 12 lowers the threshold used for the determination. (3) The target inference unit 12 makes the model used by the configuration data inference unit 30 a more accurate model. For example, when the configuration data inference unit 30 uses a parameter that specifies a model, the target inference unit 12 changes the parameter so that the model becomes more accurate. (4) The target inference unit 12 increases the scale of the model used by the configuration data inference unit 30. For example, when the configuration data inference unit 30 uses a parameter that specifies a model, the target inference unit 12 changes the parameter to a parameter of a model of a larger scale. (5) If the data is an image, the target inference unit 12 increases the size of the image to be copied into the constituent images. For example, if the data construction unit 20 uses a parameter related to image copying, the target inference unit 12 changes the parameter to a parameter that increases the size of the image to be copied into the constituent images. For example, if the data construction unit 20 uses a parameter that specifies the number of grids in the constituent images, the target inference unit 12 changes the parameter to a parameter that reduces the number of grids. (6) When a gap is provided between images in the constituent images, the target inference unit 12 widens the gap. For example, when the data construction unit 20 uses a parameter for the width of the gap, the target inference unit 12 changes the parameter to a parameter for widening the gap. (7) The target inference unit 12 increases the size of the constituted data. For example, when the data constructing unit 20 uses a parameter that specifies the size of the constituted image, the target inference unit 12 changes the parameter to a parameter that increases the size of the constituted image.

[0102] Furthermore, the information processing device 102 may adjust at least one operation in the information processing device 102 as follows, based on the situation such as the inference accuracy or inference speed in at least one of the configured data inference unit 30 and the data inference unit 40. The at least one operation in the information processing device 102 is an operation in at least one of the target inference unit 12, the data configuration unit 20, the configured data inference unit 30, and the data inference unit 40. For example, the target inference unit 12 may execute the following operation as adjustment of the inference accuracy and inference speed. Note that, in order to realize the following operation, the target inference unit 12 may change a predetermined parameter in the same manner as described above. However, in the following explanation, the explanation of the change of the parameter will be omitted. (1) The target inference unit 12 changes the model used by the configured data inference unit 30 to a large-scale model. This change improves the inference accuracy of the configured data inference unit 30. Note that if the number of inferences in the data inference unit 40 is reduced as a result of the improvement in the inference accuracy of the configured data inference unit 30, this change may improve the inference speed of the information processing device 102, although this is in balance with the inference speed of the configured data inference unit 30. (2) The subject inference unit 12 changes the model used by the data inference unit 40 to a large-scale model. This change improves the inference accuracy of the data inference unit 40. (3) The object inference unit 12 changes the model used by the data inference unit 40 to a small-scale model. This change improves the inference speed of the data inference unit 40. (4) The object inference unit 12 reduces the number of grids used by the data constructing unit 20. This change improves the inference accuracy of the constructed data inference unit 30. (5) The object inference unit 12 increases the number of grids used by the data configuration unit 20. This change improves the inference speed of the configured data inference unit 30. (6) The object inference unit 12 increases the spacing between images used by the data constructor 20. This change improves the inference accuracy of the constructed data inference unit 30. (7) The target inference unit 12 changes the value of the threshold used to determine whether an inference in the constituted data inference unit 30 has succeeded or failed. For example, when the certainty of an inference is used to determine whether an inference has succeeded or failed, increasing the certainty threshold improves the inference accuracy of the constituted data inference unit 30.

[0103] The dataset generating unit 62 generates a learning dataset using an original dataset, similarly to the dataset generating unit 60. However, when at least one of the configuration data inference unit 30 and the data inference unit 40 uses a plurality of models, the dataset generating unit 62 generates a plurality of learning datasets for generating each of the plurality of models. The dataset generating unit 62 may generate a plurality of datasets using a plurality of original datasets. Alternatively, the dataset generating unit 62 may generate a plurality of original datasets by applying a data augmentation technique to the original dataset, and generate a plurality of learning datasets using the generated plurality of original datasets. For example, the dataset generating unit 62 may use at least one of left-right inversion, up-down inversion, partial cropping, combination, enlargement / reduction, brightness adjustment, luminance adjustment, and color adjustment as a data augmentation technique. The dataset generating unit 62 may use image processing different from the above.

[0104] The dataset storage unit 52, like the dataset storage unit 50, stores a training dataset used to generate a trained model used by the information processing device 102. However, when at least one of the configuration data inference unit 30 and the data inference unit 40 uses multiple models, the dataset storage unit 52 stores multiple training datasets corresponding to the multiple models. The dataset storage unit 52 may store a training dataset acquired from an operator or the like as at least a part of the training dataset.

[0105] The model learning unit 72 generates a trained model using a training data set, similar to the model learning unit 70. However, when at least one of the configuration data inference unit 30 and the data inference unit 40 uses multiple models, the model learning unit 72 generates multiple trained models. Then, the model learning unit 72 stores the generated trained models in the model storage unit 82.

[0106] The model storage unit 82 stores the trained models used by the configuration data inference unit 30 and the data inference unit 40 for inference, similar to the model storage unit 80. However, when at least one of the configuration data inference unit 30 and the data inference unit 40 uses multiple models, the model storage unit 82 stores multiple trained models. The model storage unit 82 stores trained models generated by the model learning unit 72. However, the model storage unit 82 may store trained models acquired in advance by the information processing device 102 from an operator or the like as at least a portion of the trained models. At least a portion of the multiple trained models may be models that differ from other trained models in at least one of the model structure and the network structure. For example, at least a portion of the trained models may be models that differ from other trained models in at least a portion of the following items. (a) Training dataset, (b) network structure; (c) Hyperparameters, (d) Weight accuracy, (e) batch size; (f) Parameters related to the generation of the constituent images.

[0107] The information processing device 102 configured in this manner can improve the throughput of inference, and can switch the operation of the information processing device 102 to a more appropriate operation according to the status of the inference operation, as in the second embodiment. More specifically, the target inference unit 12 of the information processing device 102 switches the operation of the information processing device 102 to a more appropriate operation based on the status of inference in at least one of the configuration data inference unit 30 and the data inference unit 40. The status of inference may be the inference accuracy or the inference speed. However, the status of inference is not limited to these. The operation switched by the target inference unit 12 of the information processing device 102 is the operation in at least one of the target inference unit 12, the data configuration unit 20, the configuration data inference unit 30, and the data inference unit 40. The switching of the operation may be a change of a threshold value used to determine whether the inference of the target included in the configuration data in the target inference unit 12 has succeeded or failed. Alternatively, the switching of the operation may be a change of a model used for inference in at least one of the configuration data inference unit 30 and the data inference unit 40. Alternatively, the switching of operations may be a change in the size of the configuration data in the data configuration unit 20. Alternatively, the switching of operations may be a combination of at least some of the above. However, the switching of operations is not limited to these. For example, when the inference accuracy by the configuration data inference unit 30 decreases, the information processing device 102 may change the model used by the configuration data inference unit 30 to a model with higher accuracy. Based on such operations, the information processing device 102 achieves a desired inference accuracy or inference speed while improving throughput.

[0108] <Hardware configuration> Using the information processing device 100, the hardware configurations of the information processing devices 100, 101, 102, and 103 will be described. In FIG. 11, the information processing device 100 includes all the configurations. However, the configuration of the information processing device 100 is not limited to the configuration in FIG. 11. For example, the information processing device 100 may be configured by connecting devices having functions corresponding to each configuration via a predetermined network. For example, the information processing device 100 may be configured using cloud computing. Alternatively, at least a part of the multiple components in the information processing device 100 may be configured by a single piece of hardware. Alternatively, each component of the information processing device 100 may be configured by an individual hardware circuit.

[0109] Alternatively, the information processing device 100 may be realized as a computer device including a CPU, a read only memory (ROM), and a random access memory (RAM). The information processing device 100 may be realized as a computer device including a network interface circuit (NIC) in addition to the above configuration. Furthermore, the information processing device 100 may be realized as a computer device including an AI device that executes part or all of machine learning and inference.

[0110] 11 is a block diagram showing the configuration of a computer device 600, which is an example of the hardware configuration of the information processing device 100. The computer device 600 includes a CPU 610, an AI device 611, a ROM 620, a RAM 630, a storage device 640, and a NIC 650, and constitutes the computer device.

[0111] The CPU 610 reads a program from at least one of the ROM 620 and the storage device 640. The CPU 610 then controls the AI ​​device 611, the RAM 630, the storage device 640, and the NIC 650 based on the read program. The computer device 600 including the CPU 610 then controls these configurations and realizes the functions of the target inference unit 10, the data configuration unit 20, the configuration data inference unit 30, and the data inference unit 40. The computer device 600 including the CPU 610 further realizes the functions of the data set storage unit 50, the data set generation unit 60, the model learning unit 70, the model storage unit 80, and the data acquisition unit 90.

[0112] When implementing each function, the CPU 610 may use the RAM 630 or the storage device 640 as a temporary storage medium for the program. The CPU 610 may also read, using a storage medium reading device (not shown), a program contained in a storage medium 690 that stores the program in a computer-readable manner. Alternatively, the CPU 610 may receive a program from an external device (not shown) via the NIC 650, store the program in the RAM 630 or the storage device 640, and operate based on the stored program.

[0113] The AI ​​device 611 executes part or all of the machine learning and inference processes under the control of the CPU 610. For example, the AI ​​device 611 is a GPU, an FPGA, an ASIC, or an AI chip. In executing the processes, the AI ​​device 611 reads information required for executing the processes, such as data, programs, or circuit information, from the ROM 620, the RAM 630, or the storage device 640 under the control of the CPU 610.

[0114] The ROM 620 stores programs executed by the CPU 610 and fixed data. The ROM 620 may store information necessary for processing by the AI ​​device 611. The ROM 620 is, for example, a programmable ROM (P-ROM) or a flash ROM. The RAM 630 temporarily stores programs executed by the CPU 610 and data. The RAM 630 may temporarily store information necessary for processing by the AI ​​device 611. The RAM 630 is, for example, a dynamic RAM (D-RAM).

[0115] The storage device 640 stores data and programs that the computer device 600 stores long-term. The storage device 640 may store information necessary for processing by the AI ​​device 611. Alternatively, the storage device 640 may operate as the dataset storage unit 50. Alternatively, the storage device 640 may operate as the model storage unit 80. Alternatively, the storage device 640 may operate as a temporary storage device for the CPU 610. The storage device 640 is, for example, a hard disk device, a magneto-optical disk device, a solid state drive (SSD), or a disk array device.

[0116] The ROM 620 and the storage device 640 are non-volatile (non-transitory) recording media. On the other hand, the RAM 630 is a volatile (transitory) recording medium. The CPU 610 can operate based on a program stored in the ROM 620, the storage device 640, or the RAM 630. In other words, the CPU 610 can operate using a non-volatile recording medium or a volatile recording medium.

[0117] The NIC 650 relays data exchange with external devices (not shown) via a network. The NIC 650 is, for example, a LAN (Local Area Network) card. Furthermore, the NIC 650 is not limited to being wired, and may be wireless.

[0118] The CPU 610 and AI device 611 of the computer apparatus 600 configured in this manner can realize functions similar to those of the information processing device 100 based on a program, thereby improving the throughput of the inference process. Note that the information processing device 101 in FIG. 1, the information processing device 103 in FIG. 9, or the information processing device 102 in FIG. 10 may be realized using the computer apparatus 600.

[0119] <System> Using the information processing device 100, an example of a system including the information processing devices 100, 101, 102, and 103 will be described. FIG. 12 is a block diagram showing an example of a configuration of an information processing system 400 including the information processing device 100. The information processing system 400 includes the information processing device 100, a data acquisition device 200, and a recognition device 300. The information processing system 400 may include multiple devices as each device. For example, the information processing system 400 may include multiple data acquisition devices 200. Note that the information processing system 400 may include the information processing device 101, 102, or 103 instead of the information processing device 100.

[0120] The data acquisition device 200 acquires a plurality of data including at least a part of the target of inference, and outputs the data to the information processing device 100. The data acquisition device 200 may acquire an original data set, a learning data set, or a learned model used by the information processing device 100, and output the data to the information processing device 100. The data acquisition device 200 is, for example, a surveillance camera. In this case, the data acquisition device 200 outputs a captured image to the information processing device 100 as data including the target of inference. For example, the data acquisition device 200 acquires an image of a car's LP, and outputs an image of characters included in the acquired image to the information processing device 100. Alternatively, the data acquisition device 200 may be a device that stores data for inference acquired in advance.

[0121] As described above, the information processing device 100 acquires data including an inference target from the data acquisition device 200, and executes the operations described above to perform inference on the target included in the acquired data. For example, when the information processing device 100 executes an image classification task for alphanumeric characters, the information processing device 100 classifies the image acquired from the data acquisition device 200 into either the numbers 0 to 9 or the alphabets A to Z. Then, the information processing device 100 outputs an inference result for the target. For example, the information processing device 100 outputs a character class to the recognition device 300 as an inference result for the target. The information processing device 100 may output data used for inference, such as an image including characters, in accordance with the inference result.

[0122] The recognition device 300 uses the inference result acquired from the information processing device 100 to perform recognition related to the object inferred by the information processing device 100. For example, when the information processing device 100 classifies the class of alphanumeric characters in the image of the characters of the LP, the recognition device 300 recognizes the LP of a car using the classified class of alphanumeric characters. In this case, the class of alphanumeric characters is the inference result of the information processing device 100, and the recognition process of the LP is the recognition process related to the object, that is, the recognition process in the recognition device 300. Furthermore, the recognition device 300 may output the recognition result. For example, the recognition device 300 may display the recognition result on a predetermined display device. For example, when the recognition device 300 recognizes the LP, the user can determine the LP by referring to the LP displayed on the display device.

[0123] As described above, the information processing system 400 includes the information processing device 100, the data acquisition device 200, and the recognition device 300. The data acquisition device 200 outputs a plurality of data including at least a part of an object to the information processing device 100. The information processing device 100 executes inference for the object included in the data. The recognition device 300 acquires an inference result for the object from the information processing device 100, and executes recognition related to the object based on the acquired inference result. For example, the data acquisition device 200 acquires an image including characters of the LP of a car. The information processing device 100 infers a class of the image of the characters. The recognition device 300 recognizes the LP based on the inferred class of the characters. Then, the information processing device 100 in the information processing system 400 realizes inference with improved throughput. Therefore, the information processing system 400 improves the throughput of the recognition process based on the inference result of the information processing device 100.

[0124] Although the present invention has been described above with reference to the embodiment, the present invention is not limited to the above embodiment. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Explanation of symbols]

[0125] 10. Object Inference Section 11 Object Inference Section 12 Object Inference Section 20 Data Component 21 Data Component 23 Data Component 30 Configuration Data Inference Section 31 Configuration Data Inference Unit 33 Configuration Data Inference Unit 40 Data Inference Department 41 Data Inference Department 50 Data Set Storage Section 52 Data Set Storage Section 60 Dataset Generation Unit 62 Dataset Generation Unit 70 Model Learning Department 72 Model Learning Department 80 Model Storage Section 82 Model Storage Section 90 Data Acquisition Section 100 Information processing device 101 Information processing device 102 Information processing equipment 103 Information processing equipment 600 Computer equipment 610 CPU 611 AI Device 620 ROM 630 RAM 640 Storage device 650NIC 690 Recording media

Claims

1. A data construction means for generating constructed data constructed using a plurality of data, at least a part of which includes an inference target; A configuration data inference means for executing an inference on the object included in the configuration data; A data inference means for performing inference on the object included in at least a part of the data; For each of the data constituting the configuration data, it is determined whether an inference on the object included in the configuration data has succeeded or failed; As an inference for the object included in the data for which an inference for the object included in the configuration data has been successful, an inference for the object is executed based on the inference result for the object included in the configuration data and the configuration data; As an inference for the object included in the data for which inference for the object included in the configuration data has failed, an inference for the object is executed based on an inference result for the object included in the data. Object inference means and An information processing device comprising:

2. a second data constructing means for generating second constructed data constructed using a plurality of the data for which inference on the target included in the constructed data has failed; a second configuration data inference means for executing an inference on the object included in the second configuration data; Further comprising: The object inference means includes: For each of the data constituting the second constituted data, it is determined whether an inference on the target included in the second constituted data has succeeded or failed; As an inference for the object included in the data for which an inference for the object included in the second configuration data has been successful, an inference for the object is executed based on an inference result for the object included in the second configuration data and the second configuration data; As an inference for the object included in the data for which inference for the object included in the second configuration data has failed, an inference for the object is performed based on an inference result for the object included in the data. The information processing device according to claim 1 .

3. The object inference means switches the operation of at least one of the object inference means, the data configuration means, the configuration data inference means, and the data inference means based on an inference status of at least one of the configuration data inference means and the data inference means.

3. The information processing device according to claim 1 or 2.

4. The inference status is inference accuracy or inference speed, The switching of the operation is a change of a threshold value used for determining whether an inference on the object included in the configuration data in the object inference means has succeeded or failed, a change of a model used for inference in at least one of the configuration data inference means and the data inference means, a change of a size of the configuration data in the data configuration means, or a combination of these. The information processing device according to claim 3 .

5. A data set generating means for generating a configuration data learning data set for generating a configuration data inference model and a data learning data set for generating a data inference model, using a predetermined data set; A dataset storage means for storing the configuration data learning dataset and the data learning dataset; a model learning means for generating a configured data inference model by machine learning using the configured data learning dataset, and for generating a data inference model by machine learning using the data learning dataset; a model storage means for storing the configuration data inference model and the data inference model; A data acquisition means for acquiring the data; Including, The data configuration means generates the configured data composed of the data acquired from the data acquisition means, the configuration data inference means executes inference on the object included in the configuration data by using the configuration data inference model stored in the model storage means; The data inference means acquires the data specified by the data configuration means from the data acquisition means, and executes inference on the object included in the acquired data by using the data inference model stored in the model storage means.

5. The information processing apparatus according to claim 1.

6. the data is an image of two-dimensional data; The data constructing means generates a constructed image by arranging a plurality of images on a two-dimensional plane; The constituent data inference means performs an object detection task as an inference for the object included in the constituent image; The data inference means performs an image classification task as an inference for the object contained in the image.

6. The information processing apparatus according to claim 1.

7. An information processing device according to any one of claims 1 to 6, A data acquisition device that outputs a plurality of the data including at least a part of the target to the information processing device; a recognition device that acquires an inference result for the object from the information processing device and performs recognition related to the object based on the acquired inference result; An information processing system comprising:

8. An information processing device comprising: generating constructed data using a plurality of pieces of data at least a part of which includes a subject of inference; performing inference on the object included in the configuration data; performing inferences on at least some of the data with respect to the objects contained in the data; For each of the data constituting the configuration data, it is determined whether an inference on the object included in the configuration data has succeeded or failed; As an inference for the object included in the data for which an inference for the object included in the configuration data has been successful, an inference for the object is executed based on the inference result for the object included in the configuration data and the configuration data; As an inference for the object included in the data for which inference for the object included in the configuration data has failed, an inference for the object is executed based on an inference result for the object included in the data. Information processing methods.

9. An information processing device executes the method of claim 8, A data acquisition device outputs a plurality of pieces of data including at least a part of the target to the information processing device; A recognition device acquires an inference result of the object from the information processing device, and performs recognition related to the object based on the acquired inference result. Information processing methods.

10. A process of generating constituted data composed of a plurality of pieces of data at least a part of which includes a subject of inference; A process of performing inference on the object included in the configuration data; performing inferences on at least some of the data with respect to the objects contained in the data; For each of the data constituting the configuration data, it is determined whether an inference on the object included in the configuration data has succeeded or failed; As an inference for the object included in the data for which an inference for the object included in the configuration data has been successful, an inference for the object is executed based on the inference result for the object included in the configuration data and the configuration data; a process of executing an inference for the object included in the configuration data based on an inference result for the object included in the data as an inference for the object included in the configuration data that has failed; A program that causes a computer to execute the following.

Images