Information processing device, information processing system, information processing method, and program

Abstract

In order to improve inference throughput, an information processing device according to the present invention includes: a primary inference means for inferring, as a primary inference, a first target object by applying, to a first trained model, inference target data in which at least some data includes the first target object; a data aggregation means for categorizing, into any one group on the basis of attributes of the first target object, the first target object inferred in the primary inference, generating aggregated data which is data in an amount smaller than the inference target data for each group, and generating a correspondence relationship between a position of the first target object in the inference target data and a position of the first target object in the aggregated data; a plurality of secondary inference means for inferring, as a secondary inference, the first target object by applying the aggregated data of different groups to a second trained model; and an object inference means for using the first target object in the second inference results and the correspondence relationship to infer the first target object in the inference target data.

Description

[Technical field]

[0001] The present invention relates to inference using machine learning. [Background technology]

[0002] Non-Patent Documents 1 to 3 disclose techniques for an object detection task using deep learning. Non-Patent Document 4 discloses a technique for a posture estimation task that estimates the posture of a person included in an image. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", [online], 6 January 2016, Cornel University, [Retrieved July 20, 2021], Internet<URL:https: / / arxiv.org / abs / 1506.01497> [Non-Patent Document 2] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C. Berg, "SSD: Single Shot MultiBox Detector", [online], 29 December 2016, Cornel University, [Retrieved July 20, 2021, Internet,<URL:https: / / arxiv.org / abs / 1512.02325> [Non-Patent Document 3] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollar, "Focal Loss for Dense Object Detection", [online], 7 February 2018, Cornel University, [Retrieved July 20, 2021], Internet,<URL:https: / / arxiv.org / abs / 1708.02002> [Non-Patent Document 4] Yadong Pan, Ryo Kawai, Noboru Yoshida, Hiroo Ikeda, Shoji Nishimura, “Training Physical and Geometrical Mid-Points for Multi-person Pose Estimation and Human Detection Under Congestion and Low Resolution”, June 21, 2020, The National Center for Biotechnology Information, [Retrieved September 15, 2021], Internet<URL: https: / / pubmed.ncbi.nlm.nih.gov / 33063050 / > Summary of the Invention [Problem to be solved by the invention]

[0004] Inference processing using machine learning, particularly object detection task and pose estimation task processing, is a process that has a high calculation load and takes a long time to process. Therefore, it is desired to improve the throughput of inference processing in object detection tasks and pose estimation tasks. Non-Patent Documents 1 to 4 do not disclose techniques related to improving the inference throughput. An object of the present invention is to provide an information processing device or the like that improves the inference throughput of a target object. [Means for solving the problem]

[0005] According to one aspect of the present invention, there is provided an information processing device comprising: a primary inference means for inferring the first target object by applying inference target data, at least a portion of which includes the first target object, to a first trained model as a primary inference; a data aggregation means for classifying a first target object inferred in the primary inference into one of groups based on the attributes of the first target object, generating aggregated data for each group that is data with a smaller amount than the inference target data, and generating a correspondence relationship between the position of the first target object in the inference target data and the position of the first target object in the aggregated data; a plurality of secondary inference means for inferring a first target object by applying aggregated data of different groups to a second trained model as a secondary inference; an object inference means for inferring a first object in the inference target data by using a first object in the result of the secondary inference and the correspondence relationship; Includes.

[0006] An information processing system according to one embodiment of the present invention includes: The above information processing device; A data acquisition device that outputs inference target data to an information processing device; a display device that acquires the inference result from the information processing device and displays the acquired inference result; Includes.

[0007] An information processing method according to one aspect of the present invention includes: As a first inference, inference target data, at least a portion of which includes a first target object, is applied to a first trained model to infer the first target object; classifying the first target object inferred in the primary inference into one of the groups based on the attributes of the first target object, generating aggregated data for each group, which is data having a smaller quantity than the inference target data, and generating a correspondence relationship between the position of the first target object in the inference target data and the position of the first target object in the aggregated data; As a secondary inference, the aggregated data of the different groups is applied to a second trained model to infer the first target object; The first object in the result of the secondary inference is used to infer the first object in the inference target data by using the corresponding relationship.

[0008] An information processing method according to one aspect of the present invention includes: An information processing device executes the above information processing method, The data acquisition device outputs inference target data to the information processing device; The display device acquires the result of inference from the information processing device and displays the acquired result of inference.

[0009] A recording medium according to one embodiment of the present invention includes: A process of applying inference target data, at least a portion of which includes a first target object, to a first trained model as a first inference to infer the first target object; A process of classifying the first target object inferred in the primary inference into one of the groups based on the attributes of the first target object, generating aggregated data for each group, which is data with a smaller amount than the inference target data, and generating a correspondence relationship between the position of the first target object in the inference target data and the position of the first target object in the aggregated data; As a secondary inference, a process of inferring a first target object by applying the aggregated data of the different groups to a second trained model; A process of inferring a first target object in the inference target data by using the first target object in the result of the secondary inference and the correspondence relationship. A program that causes a computer to execute the above is recorded. Effect of the Invention

[0010] According to the present invention, it is possible to obtain an effect of improving the throughput of inference of a target object. [Brief description of the drawings]

[0011] [Figure 1] FIG. 11 is a block diagram showing an example of a configuration of an information processing device according to a second embodiment. [Diagram 2]FIG. 11 is a flow diagram showing an example of an operation for generating a trained model in an information processing device according to the second embodiment. [Diagram 3] FIG. 11 is a flow diagram showing an example of an operation of inferring a target object in the information processing device according to the second embodiment. [Figure 4] FIG. 11 is a flow diagram illustrating an example of post-processing in a data aggregation unit. [Diagram 5] FIG. 13 is a diagram showing an example of a result of primary inference. [Figure 6] FIG. 11 is a diagram illustrating an example of aggregated data. [Figure 7] FIG. 13 is a diagram showing an example of a result of secondary inference. [Figure 8] FIG. 13 is a diagram showing an example of inference of a target object in inference target data. [Figure 9] FIG. 2 is a block diagram showing an example of a hardware configuration of an information processing device. [Figure 10] FIG. 1 is a block diagram showing an example of a configuration of an information processing system including an information processing device. [Figure 11] FIG. 13 is a block diagram illustrating an example of a configuration of an information processing device according to a third embodiment. [Figure 12] FIG. 11 is a flow diagram showing an example of a model switching operation in the information processing device according to the third embodiment. [Figure 13] 1 is a block diagram showing an example of a configuration of an information processing device according to a first embodiment. [Figure 14] FIG. 11 is a block diagram showing an example of the configuration of an information processing device that executes secondary inference for each group according to the second embodiment. [Figure 15] 15 is a flow diagram showing an example of an operation of inferring a target object in the information processing device of FIG. 14. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Each drawing is for explaining an embodiment of the present invention. However, each embodiment is not limited to the description of each drawing. In addition, each embodiment can be appropriately combined.

[0013] First Embodiment FIG. 13 is a block diagram showing an example of the configuration of an information processing device 1C according to the first embodiment of the present invention. The information processing device 1C includes an object inference unit 10D, a primary inference unit 20D, a data aggregation unit 30D, and a plurality of secondary inference units 40D. The primary inference unit 20D applies inference target data, at least a part of which includes a first target object, to a first learned model as a primary inference to infer the first target object. The data aggregation unit 30D classifies the first target object inferred in the primary inference into one of the groups based on the attributes of the first target object. Then, the data aggregation unit 30D generates aggregated data, which is data with a smaller amount than the inference target data for each group. Then, the data aggregation unit 30D generates a correspondence relationship between the position of the first target object in the inference target data and the position of the first target object in the aggregated data. The secondary inference unit 40D applies aggregated data of different groups to a second learned model to infer the first target object as a secondary inference. The object inference unit 10D infers the first target object in the inference target data by using the first target object in the result of the secondary inference and the correspondence relationship.

[0014] The information processing device 1C thus configured generates aggregated data that aggregates the inference target data for each group of the first target object classified based on the attributes of the first target object. Then, the information processing device 1C infers the target object in the inference target data using the result of the secondary inference using the aggregated data for each group and the correspondence between the inference target data of the target object and the position of the aggregated data. The aggregated data includes the target object in the original data and is data with a small amount of data. Therefore, the information processing device 1C improves the throughput of inference of the target object. Furthermore, the information processing device 1C executes secondary inference on aggregated data aggregated for each of a plurality of groups classified based on the attributes of the first target object. That is, the information processing device 1C executes secondary inference corresponding to the attributes of the first target object. For example, as will be described in detail later, for groups with low inference difficulty, the information processing device 1C can improve the throughput of inference while maintaining accuracy by using secondary inference that uses a learned model or inference parameter that is not highly accurate but is lightweight and fast as the secondary inference. On the other hand, for a group with high inference difficulty, the information processing device 1C can perform inference while avoiding deterioration of accuracy by using a secondary inference that uses a highly accurate trained model or inference parameters. In this way, the information processing device 1C improves the throughput of inference of a target object while maintaining accuracy.

[0015] <Second embodiment> One of the main tasks using machine learning is the object detection task in an image. The object detection task is a task to generate a list of pairs of positions and classes (types) of target objects present in an image. In recent years, object detection tasks using deep learning, especially in machine learning, have been widely used. In the object detection task, a group of images for learning and information on target objects in each image are given as correct answer data in the learning phase of machine learning. The information on the target objects is selected according to the specifications of the object detection task. For example, the information on the target object includes the coordinates of the four vertices of a rectangular area in which the target object is reflected (bounding box (BB)) and the class of the target object. Note that in the following description, the BB and class are used as an example of the information on the target object. Then, the object detection task uses a group of images for learning and the information on the target object to generate a trained model as a result of machine learning using, for example, deep learning. In the detection phase, the object detection task applies the trained model to an image including the target object to infer the target object included in the image, and outputs a pair of the BB and the class for each target object included in the image. Note that the object detection task may output an evaluation result of the object detection result together with the BB and the class. As the evaluation result, confidence is often used, but the evaluation result is not limited to this, and other values ​​may be used.

[0016] For example, a surveillance system for people and cars can be built using an object detection task. For example, a user of the system inputs an image from a surveillance camera into the object detection task. The object detection task detects the positions and classes of people and cars in the image. The user can then use the positions and classes of people and cars detected by the object detection task to determine the people and cars appearing in the surveillance camera image. The object detection task may also be used in combination with other recognition processes. For example, a license plate recognition system can be built by combining an object detection task that detects license plates of cars with an optical character recognition (OCR) process that recognizes characters on the detected license plates.

[0017] Another task is a posture estimation task that estimates the posture of a person included in an image. The posture estimation task is composed of, for example, a process of extracting keypoints such as eyes, nose, elbows, and knees, and a process of combining the keypoints to generate the posture of the person. The posture estimation task may, for example, be, but not limited to, a case in which deep learning is used for the keypoint extraction process that extracts the feature points, and non-deep learning is used for the posture generation process. In the keypoint extraction process, in the inference phase, a trained model is applied to an image including a target to infer keypoints such as eyes and nose included in the image. Then, in the keypoint extraction process, for each inferred keypoint, the coordinates and class of the keypoint and the certainty of the inference are output.

[0018] High throughput is required for the processing of object detection tasks and pose estimation tasks. For example, in order to prevent oversight, a surveillance camera needs to capture images at a relatively high frame rate. In particular, when targeting a fast-moving object such as a car, the surveillance camera is required to operate at a frame rate of, for example, 10 fps (frames per second) or more. Therefore, the object detection task and the like need to detect objects in images captured by the surveillance camera at a high frame rate.

[0019] Furthermore, in a system for detecting an object in an image captured by a surveillance camera, a configuration is assumed in which the surveillance camera or a device in the vicinity of the surveillance camera, i.e., an edge, executes inference processing for an object detection task. However, in the edge, there are often restrictions on installation space, cooling, power, etc., and only limited computing resources are available. Processing such as object detection tasks requires a certain degree of high throughput even in such an edge environment.

[0020] However, inference processing using machine learning, particularly object detection task and pose estimation task processing, is a process that requires a high computation load and a long processing time. As a result, object detection tasks and pose estimation tasks generally tend to have low throughput performance. Therefore, it is desirable to improve the throughput of inference processing in object detection tasks and pose estimation tasks.

[0021] Therefore, the second embodiment improves the inference throughput by using a configuration described in detail below. The configuration of the second embodiment will be described with reference to the drawings. FIG. 1 is a block diagram showing an example of the configuration of an information processing device 1 according to the second embodiment. Note that the following will be used as an example in the following description including this embodiment. (1) Data to be inferred: Images (still images or videos) taken by a camera, etc. (2) Inference task: an object detection task or a pose estimation task in an image; (3) Inference result: a set of “BB or Keypoint, class, and confidence” (4) Target object: the vehicle's license plate (LP) or posture.

[0022] However, these do not limit the present embodiment. For example, the present embodiment may use data other than an image as data to be processed. For example, the present embodiment may use depth data acquired using a depth sensor as data to be processed. Alternatively, the present embodiment may use three-dimensional data acquired using a LIDAR (Light Detection and Ranging (LIDAR)) as data to be processed. Alternatively, the result of inference in the present embodiment is not limited to a set of "BB or Keypoint, class, and confidence". For example, the present embodiment may infer a target object included in the BB. In the following description, for convenience of explanation, the set of "BB, class, and confidence" may be simply called "BB".

[0023] The configuration of the second embodiment will be described with reference to the drawings. FIG. 1 is a block diagram showing an example of the configuration of an information processing device 1 according to the second embodiment. The information processing device 1 includes an object inference unit 10, a primary inference unit 20, a data aggregation unit 30, a secondary inference unit 40, a data storage unit 50, a data generation unit 60, a model generation unit 70, a model storage unit 80, and a data acquisition unit 90. The number of components and the connection relationship shown in FIG. 1 are an example. For example, the information processing device 1 may include a plurality of data acquisition units 90. The information processing device 1 may be configured using a computer device including a CPU (Central Processing Unit), a main memory, and a secondary storage device. In this case, the components of the information processing device 1 shown in FIG. 1 indicate functions realized using a CPU or the like. The hardware configuration will be further described later.

[0024] In the information processing device 1, the configuration for controlling the operation of each component is arbitrary. For example, the information processing device 1 may include a control unit (not shown) that controls each component. Alternatively, a predetermined configuration may control the operation of components such as the primary inference unit 20, the data aggregation unit 30, and the secondary inference unit 40. Alternatively, for example, the object inference unit 10 may control the operation of components such as the primary inference unit 20, the data aggregation unit 30, and the secondary inference unit 40. In this manner, a predetermined configuration may control other configurations. Alternatively, each component may independently acquire data from other components and operate. Alternatively, each component may start operating when it acquires data from other components. Therefore, in the following description, a description regarding the control of the operation of the components will be omitted unless particularly necessary.

[0025] (1) Data Acquisition Unit 90 The data acquisition unit 90 acquires data including an object to be inferred from a specified device. Hereinafter, the object to be inferred is referred to as a "target object." Moreover, data including the target object is referred to as "inference target data." At least a portion of the inference target data needs to include the target object. In other words, at least a portion of the inference target data does not need to include the target object. The device from which the data acquisition unit 90 acquires the inference target data is arbitrary. For example, the data acquisition unit 90 may acquire an image from a surveillance camera as the inference target data.

[0026] (2) Primary reasoning section 20 The primary inference unit 20 infers a target object in the inference target data. Hereinafter, the inference in the primary inference unit 20 is referred to as "primary inference". In addition to the target object, the primary inference unit 20 may infer an object that is in a predetermined positional relationship with the target object as a target of the primary inference. The object that is in a predetermined positional relationship with the target object is, for example, a peripheral object of the LP, but is not limited thereto. Hereinafter, the object that is in a predetermined positional relationship with the target object is referred to as a "sub-target object". In the following description, the primary inference unit 20 infers the target object and the sub-target object. However, this does not limit the operation of the primary inference unit 20. The primary inference unit 20 does not need to infer the sub-target object. In addition, in the following description, the target object of inference in the information processing device 1 may be referred to as a "first target object", and the sub-target object may be referred to as a "second target object".

[0027] The predetermined positional relationship is arbitrary. The operator may set the positional relationship of the sub-target object in accordance with the target object. For example, one example of the positional relationship is that "the target object is contained in the sub-target object" or "the sub-target object contains the target object." For example, when the target object is the LP of a vehicle, the sub-target object is the front and rear surfaces of the vehicle that contain the LP. In the following, the front and rear surfaces of the vehicle that contain the LP may be collectively referred to as the "vehicle front and rear surfaces." However, this does not limit the positional relationship in the second embodiment.

[0028] The primary inference unit 20 uses machine learning in the primary inference. In detail, the primary inference unit 20 infers a target object and a sub-target object in the inference target data using a trained model generated using machine learning stored in the model storage unit 80. Hereinafter, the trained model used by the primary inference unit 20 is referred to as a "trained model for primary inference" or a "first trained model." The primary inference unit 20 may modify the inference target data when applying the inference target data to the trained model for primary inference. For example, when the inference target data is an image, the primary inference unit 20 may change the size or aspect ratio of the image.

[0029] (3) Data Aggregation Unit 30 The data aggregating unit 30 executes the process described below using the target object included in the result of the primary inference. However, when the primary inference unit 20 infers a sub-target object in addition to the target object, the data aggregating unit 30 may use the sub-target object in addition to the target object. In the following description, the data aggregating unit 30 uses the target object and the sub-target object. However, the data aggregating unit 30 does not have to use the sub-target object. In this case, the data aggregating unit 30 may omit the operations related to the sub-target object in the following description.

[0030] The data aggregating unit 30 performs a predetermined process on the target object included in the result of the primary inference. For example, the predetermined process is discarding the BB, filtering the BB, or adjusting the BB. Hereinafter, this process is referred to as "post-processing". The data aggregating unit 30 may perform post-processing on some of the target objects included in the result of the primary inference, or may perform post-processing on all of the target objects. Furthermore, the data aggregating unit 30 may perform post-processing on some or all of the sub-target objects included in the result of the primary inference. In the following description, as an example, the data aggregating unit 30 performs post-processing on all target objects and sub-target objects. However, the post-processing in the data aggregating unit 30 is not limited to this.

[0031] Furthermore, the data aggregation unit 30 generates data that includes the regions of the target object and the regions of the sub-target object in the inference target data and has a smaller amount of data than the inference target data. Hereinafter, the data generated as data with a smaller amount of data than the inference target data will be referred to as "aggregated data." Hereinafter, this process will be referred to as "aggregation." In other words, "aggregation" refers to extracting or duplicating regions that include the target object and sub-target objects in the result of the primary inference from the inference target data, collecting the extracted or duplicated regions, and generating aggregated data that is a smaller amount of data than the inference target data.

[0032] In the aggregated data, the quantity of data that is less than the inference target data is arbitrary. The operator may decide according to the target object and the inference operation. For example, when the number of data is used as the quantity of data, the aggregated data is data with a smaller quantity of data than the inference target data. For example, when the data is a still image, the number of data is the number of still images. Alternatively, when the data is a video of a fixed duration, the number of data is, for example, the number of videos. When the amount of data is used as the quantity of data, the aggregated data is data with a smaller quantity of data than the inference target data. The amount of data is, for example, capacity or area. For example, when the data is a still image, the amount of data is the amount of data of the entire still image. Alternatively, when the data is a video, the amount of data is the duration of the video.

[0033] Furthermore, in generating the aggregated data, the data aggregating unit 30 generates a correspondence relationship between the positions of the target object and the sub-target object in the inference target data and the positions of the target object and the sub-target object in the aggregated data. Hereinafter, the correspondence relationship between the positions of the target object and the sub-target object in the inference target data and the positions of the target object and the sub-target object in the aggregated data is referred to as an "aggregated correspondence relationship." For example, when inferring BB, the data aggregating unit 30 generates, as the aggregated correspondence relationship, a coordinate transformation between the position, orientation, and size of the target object BB and the sub-target object BB in the inference target data and their respective positions, orientations, and sizes in the aggregated data.

[0034] In addition, when the data is an image, the data aggregating unit 30 may generate an image in the aggregated data having a size different from that of the image in the inference target data. For example, an operator of the information processing device 1 may set the size of the image in the aggregated data to the data aggregating unit 30 based on the size of the inference target data, the performance of the hardware of the information processing device 1, and the processing performance and calculation accuracy of the trained model used for inference. The aggregated data is data used for inference by the secondary inference unit 40 described later. Moreover, the inference target data is data used for inference by the primary inference unit 20. That is, the information processing device 1 may use data of different sizes in the primary inference unit 20 and the secondary inference unit 40. For example, when the data is an image, the information processing device 1 may use images of different sizes in the primary inference unit 20 and the secondary inference unit 40.

[0035] Next, the details of post-processing and aggregation in the data aggregating unit 30 will be described. Note that "post-processing" and "aggregation" in the data aggregating unit 30 are collectively referred to as "data aggregation." In other words, the "data aggregation" operation includes the "post-processing" operation before the generation of aggregated data and the "aggregation" operation. However, the data aggregating unit 30 does not have to execute "post-processing." In this case, the data aggregation becomes the "aggregation" operation in the following description.

[0036] (3-1) Post-processing The data aggregating unit 30 executes post-processing described below before generating aggregated data. For example, the data aggregating unit 30 discards unnecessary sub-target object regions from the result of the primary inference using the positional relationship between the target object and the sub-target object and the result of the primary inference. For example, the data aggregating unit 30 discards sub-target objects that are included in the result of the primary inference and have a target object in a predetermined positional relationship with the sub-target object included in the result of the primary inference. Alternatively, the data aggregating unit 30 may discard the BB of the sub-target object corresponding to the target object inferred as the result of the primary inference. In other words, the data aggregating unit 30 uses, as the target of aggregation, sub-target objects that are not included in the result of the primary inference and have a target object in a predetermined positional relationship with the sub-target object included in the result of the primary inference. The reason for using this operation will be explained using "positional relationship in which the sub-target object includes the target object" and "BB".

[0037] The BB of the target object included in the result of the primary inference is included in the aggregated data in the aggregation operation described later. Therefore, even if the BB of the sub-target object including the target object included in the result of the primary inference is not included in the aggregated data, the BB of the target object is included in the aggregated data. Furthermore, if the BB of the sub-target object is included in the aggregated data, the aggregated data will include a duplicated BB as the BB of the target object. This will result in an unnecessary load being added in the secondary inference. Therefore, the data aggregation unit 30 discards such a sub-target object. As a result, the information processing device 1 can improve the throughput compared to the case where the BB of the sub-target object is used. On the other hand, the BB of the sub-target object not including the target object in the result of the primary inference may include a target object that was not inferred in the primary inference. Therefore, if the BB of such a sub-target object is included in the aggregated data, the information processing device 1 can improve the accuracy of the secondary inference.

[0038] In this way, the data aggregating unit 30 discards, among the sub-target objects, a sub-target object whose target object is included in the result of the primary inference. In other words, the data aggregating unit 30 uses, as a target for aggregation, a sub-target object whose target object in a predetermined positional relationship with the target object is not included in the result of the primary inference, among the sub-target objects in a predetermined positional relationship with the target object. As a result of this operation, the data aggregating unit 30 can reduce the amount of aggregated data generated while improving the inference accuracy in the secondary inference. As a result, the information processing device 1 can improve the throughput and the inference speed.

[0039] For example, when the positional relationship of the sub-target object is "the vehicle front and rear surfaces, which are the sub-target object, include the LP, which is the target object," the data aggregating unit 30 discards the BBs of the vehicle front and rear surfaces that include the LP included in the result of the primary inference within the BBs of the vehicle front and rear surfaces included in the result of the primary inference. In other words, the data aggregating unit 30 aggregates the BBs of the vehicle front and rear surfaces that do not have an inferred LP within their regions among the BBs of the vehicle front and rear surfaces included in the result of the primary inference. In this way, the data aggregating unit 30 generates aggregated data so as not to include the result of the primary inference in which both the target object and the sub-target object are inferred.

[0040] The positional relationship between the target object and the sub-target object is not limited to the above and is arbitrary. For example, the data aggregating unit 30 determines the positional relationship between the target object and the sub-target object using the positional relationship set by the operator. Note that the data aggregating unit 30 may use a learned model that determines the positional relationship between the target object and the sub-target object, which is stored in a device (not shown) or the like, in determining the positional relationship between the target object and the sub-target object.

[0041] The data aggregating unit 30 may perform an operation different from discarding the sub-target object. For example, when a target object is detected and a sub-target object corresponding to the target object is detected, the data aggregating unit 30 may update the confidence of the BB of the target object without discarding the BB of the sub-target object. For example, when a target object is detected and a sub-target object corresponding to the target object is detected, the data aggregating unit 30 may add or multiply the confidence of the BB of the target object by a predetermined value, or may change the value of the confidence to a predetermined value.

[0042] The data aggregating unit 30 may filter the result of the primary inference as a post-processing before generating aggregated data. The target of filtering is arbitrary. For example, the data aggregating unit 30 may filter the BB of the result of the primary inference using at least one of the confidence corresponding to the BB, the size or aspect ratio of the BB, and the position of the BB in the image. More specifically, for example, the data aggregating unit 30 may filter the BB based on a comparison between a predetermined threshold and an attribute value of the BB. Note that the attribute value is, for example, the size or aspect ratio. For example, the data aggregating unit 30 may discard a BB with a size or aspect ratio that is inappropriate for inference. Note that the threshold may not be one, but may be multiple. For example, the threshold may be different for each class related to the BB. Alternatively, the threshold may be different depending on whether a sub-target object corresponding to the target object is detected or not.

[0043] Alternatively, the data aggregation unit 30 may adjust at least one of the size and aspect ratio of the acquired BB. The position and size of the BB included in the result of the primary inference may differ from the correct position and size. For example, the BB size may be smaller than the actual target object. In particular, when the trained model used in the primary inference is small and lightweight, the inferred BB tends to be smaller than the actual target object.

[0044] If the BB, whose size is smaller than the actual target object, is used as is, the data aggregating unit 30 generates aggregated data including the target object with a missing part. As a result, the secondary inference unit 40, which will be described later, will perform inference using aggregated data including the target object with a missing part. As a result, the possibility that the result of inference in the secondary inference unit 40 will be incorrect, that is, the possibility that the inference accuracy will be low, increases. Therefore, as a post-processing, the data aggregating unit 30 may correct the size of the BB by adding or multiplying a predetermined value to at least one of the height and width of the BB of the target object and the BB of the sub-target object. Alternatively, if the aspect ratio of the BB deviates from a predetermined range, the data aggregating unit 30 may correct at least one of the height and width of the BB so that the aspect ratio falls within the predetermined range. For example, the data aggregating unit 30 may use different values ​​as the amount of correction of the BB, corresponding to at least one of the BB size, class, position in the image, and relative position with respect to a predetermined object in the image. Alternatively, in the case of a BB of a target object, the data aggregating unit 30 may use different values ​​as the amount of correction of the BB depending on whether a sub-target object corresponding to the target object is detected or not, or depending on the size of the BB of the corresponding sub-target object, etc. In these cases, the operator may set in advance the respective values ​​used by the data aggregating unit 30.

[0045] It should be noted that the data aggregating unit 30 does not have to execute "post-processing". However, whether or not post-processing is executed, the data aggregating unit 30 executes the same operation as the aggregation operation described below. Therefore, for convenience of explanation, in the following explanation, the operation described below, including the case where post-processing is not executed, will be described as the operation "after post-processing". In other words, when post-processing is not executed, the information processing device 1 may operate using the result of the primary inference without executing post-processing as the operation "after post-processing" in the following explanation.

[0046] (3-2) Aggregation The data aggregating unit 30 generates aggregated data that collects the regions of the target object and sub-target objects in the result of the primary inference after post-processing. In detail, the data aggregating unit 30 extracts or copies the regions of the target object and the sub-target object in the result of the primary inference after post-processing of the inference target data, and generates aggregated data that collects the extracted or copied regions.

[0047] The data aggregating unit 30 may generate one aggregated data from multiple inference target data, or may generate multiple aggregated data from multiple inference target data. However, even in the case of multiple aggregated data, the amount of aggregated data is less than the amount of inference target data. If the inference target data is an image, the data aggregating unit 30 may provide a predetermined gap between the images when generating the aggregated data. As already explained, the data aggregating unit 30 may perform aggregation without performing post-processing. In this case, the data aggregating unit 30 performs aggregation using the result of the primary inference.

[0048] Furthermore, the data aggregating unit 30 may combine at least one of duplicated primary inference results and adjacent primary inference results into one in the primary inference results after post-processing of the inference target data, and then aggregate them. For example, the data aggregating unit 30 may aggregate duplicated BBs and BBs within a predetermined distance into a BB that includes all of these BBs. Through such an operation, the information processing device 1 can reduce the possibility of double detection of the same object and the possibility of overlooking a target object in the secondary inference.

[0049] The data aggregating unit 30 may change the size of the target object and sub-target objects in the result of the primary inference after extraction or copying and post-processing before aggregating. The size change may be, for example, enlargement or reduction. When the size of the target object and sub-target object is reduced, the amount of aggregated data generated is reduced. As a result, the possibility of shortening the time of the secondary inference, that is, the possibility of improving the throughput, increases. On the other hand, when the size of the target object and sub-target object is increased, the size of the target object etc. in the generated aggregated data, that is, the size of the target object etc. in the secondary inference, increases. As a result, the possibility of improving the inference accuracy of the secondary inference increases.

[0050] The method of changing the size of the target object and the sub-target object is arbitrary. For example, the data aggregating unit 30 may determine whether to change the size and, if so, how to change it based on the size of the target object and the sub-target object in the result of the primary inference after post-processing. Alternatively, the data aggregating unit 30 may determine whether to change the size and, if so, how to change it using the result of the primary inference.

[0051] The data aggregating unit 30 may execute a predetermined image processing in changing the size. The predetermined image processing in this case is, for example, a complementation process. Alternatively, the data aggregating unit 30 may execute a predetermined image processing in addition to or instead of changing the size. The predetermined image processing in this case is, for example, a brightness adjustment, a luminance adjustment, a color adjustment, a contrast adjustment, a geometric correction, or a combination of some or all of these. Furthermore, the data aggregating unit 30 may adjust parameters related to the image processing in response to the result of the primary inference.

[0052] Furthermore, the data aggregating unit 30 generates a correspondence relationship between the area of ​​the target object in the inference target data and the area of ​​the target object in the aggregated data as the aggregated correspondence relationship. Note that the area in this case is, for example, the position, orientation, and size of the BB. When aggregating the area of ​​the sub-target object, the data aggregating unit 30 similarly generates an aggregated correspondence relationship that is a correspondence relationship between the area of ​​the sub-target object in the inference target data and the area of ​​the sub-target object in the aggregated data for the area of ​​the sub-target object.

[0053] (4) Secondary inference section 40 The secondary inference unit 40 infers a target object in the aggregated data generated by the data aggregating unit 30. Hereinafter, the inference in the secondary inference unit 40 is referred to as "secondary inference". The secondary inference unit 40 uses machine learning in the secondary inference. In detail, the secondary inference unit 40 infers a target object in the aggregated data using a trained model stored in the model storage unit 80. Hereinafter, the trained model used by the secondary inference unit 40 is referred to as a "trained model for secondary inference" or a "second trained model". For example, the secondary inference unit 40 uses the trained model for secondary inference to infer a set of "BB, class, and confidence" of a target object included in the aggregated data as a result of secondary inference.

[0054] The secondary inference unit 40 may change the size of the aggregated data when applying the aggregated data to the trained model for secondary inference. For example, if the aggregated data is an image, the secondary inference unit 40 may change the size or aspect ratio of the image in the aggregated data before applying it to the trained model for secondary inference.

[0055] (5) Data generation unit 60 The data generating unit 60 generates a learning dataset to be used for machine learning using a predetermined dataset. Hereinafter, the predetermined dataset is referred to as an "original learning dataset." Any data may be included in the original learning dataset as long as it corresponds to the machine learning technique. For example, the original learning dataset is a group of images including correct answer data related to the target object. The source of the original learning dataset is any. For example, an operator may store the original learning dataset in the data storage unit 50 in advance. Alternatively, the data generating unit 60 may acquire the original learning dataset from a device not shown in the figure in generating the learning dataset.

[0056] In detail, the data generating unit 60 generates a training dataset used in machine learning to generate a trained model for primary inference. The trained model for primary inference is used in the primary inference unit 20. Hereinafter, the training dataset used in machine learning to generate a trained model for primary inference used by the primary inference unit 20 is referred to as a "primary inference training dataset" or a "first training dataset". Furthermore, the data generating unit 60 generates a training dataset used in machine learning to generate a trained model for secondary inference. The trained model for secondary inference is used in the secondary inference unit 40. Hereinafter, the training dataset used in machine learning to generate a trained model for secondary inference used by the secondary inference unit 40 is referred to as a "secondary inference training dataset" or a "second training dataset".

[0057] For example, the data generating unit 60 generates correct answer data of the target object and the sub-target object based on the correct answer data of the target object included in the original training dataset and information on the positional relationship between the target object and the sub-target object. Then, the data generating unit 60 generates a primary inference training dataset by adding the generated correct answer data to the original training dataset. For example, the data generating unit 60 generates a BB that is larger than the BB of the LP included in the original training dataset by multiplying the height and width by a predetermined value. This BB is correct answer data of the front and rear surfaces of the vehicle. Then, the data generating unit 60 adds the generated correct answer data of the front and rear surfaces of the vehicle to the original training dataset to generate a primary inference training dataset.

[0058] Alternatively, the data generation unit 60 applies a predetermined trained model to the original training data set as the correct answer data of the LP to infer the BB of the LP. Then, the data generation unit 60 generates a BB that is larger than the inferred BB of the LP by multiplying the height and width by a predetermined value as the correct answer data of the front and rear surfaces of the vehicle. Then, the data generation unit 60 adds the generated LP and the correct answer data of the front and rear surfaces of the vehicle to the original training data set to generate a primary inference training data set. Alternatively, the data generation unit 60 may apply a predetermined trained model to the original training data set as the correct answer data to infer the BB of the LP and the BB of the front and rear surfaces of the vehicle, and add the inferred BB to the original training data set to generate a primary inference training data set.

[0059] Alternatively, for example, the data generating unit 60 uses the data aggregating unit 30 to generate aggregated data from the supervised data of the original training dataset. Then, the data generating unit 60 adds the supervised data and the aggregated data to the original training dataset to generate a secondary inference training dataset. Note that the data generating unit 60 may generate the secondary inference training dataset by using a primary inference training dataset instead of the original training dataset. Alternatively, the data generating unit 60 may generate the secondary inference training dataset by using a dataset in which the primary inference training dataset is added to the original training dataset.

[0060] When the primary inference unit 20 and the secondary inference unit 40 use the same trained model, the data generating unit 60 may generate one training data set as the primary inference training data set and the secondary inference training data set. Alternatively, the data generating unit 60 may generate the primary inference training data set and the secondary inference training data set using different original training data sets.

[0061] The data generation unit 60 then stores the generated primary inference learning data set and secondary inference learning data set in the data storage unit 50. However, the data generation unit 60 may acquire at least one of the primary inference learning data set and secondary inference learning data set from an external device (not shown). In this case, the data generation unit 60 stores the acquired primary inference learning data set or secondary inference learning data set in the data storage unit 50.

[0062] (6) Data storage unit 50 The data storage unit 50 stores a primary inference learning dataset and a secondary inference learning dataset. Furthermore, the data storage unit 50 may store an original learning dataset. The source of the original learning dataset is arbitrary. For example, an operator may store the original learning dataset in the data storage unit 50 in advance.

[0063] The information processing device 1 may use an external storage device (not shown) as the data storage unit 50. In this case, the information processing device 1 may not include the data storage unit 50 as a physical configuration. Alternatively, the information processing device 1 may acquire at least one of a primary inference learning dataset and a secondary inference learning dataset generated in a device (not shown) or the like, and store it in the data storage unit 50. When acquiring both the primary inference learning dataset and the secondary inference learning dataset, the information processing device 1 may not include the data generation unit 60.

[0064] (7) Model generation unit 70 The model generation unit 70 performs machine learning on a predetermined model using a data set for primary inference learning stored in the data storage unit 50 to generate a trained model for primary inference used by the primary inference unit 20, that is, a first trained model. Furthermore, the model generation unit 70 performs machine learning on a predetermined model using a data set for secondary inference learning stored in the data storage unit 50 to generate a trained model for secondary inference used by the secondary inference unit 40, that is, a second trained model. Note that the source of the model used to obtain the trained model is arbitrary. For example, an operator may store a model in the model storage unit 80 in advance. Alternatively, the model generation unit 70 may obtain a model used to generate the trained model from a device not shown in the figure in generating the trained model.

[0065] The model generation unit 70 may generate models having the same model type, network structure, and size as the trained model for primary inference and the trained model for secondary inference, or may generate different models. For example, the model generation unit 70 may generate the same model as the trained model for primary inference and the trained model for secondary inference. That is, the model generation unit 70 may generate one model as the trained model for primary inference and the trained model for secondary inference. For example, when the primary inference unit 20 and the secondary inference unit 40 use the same trained model, the model generation unit 70 may generate one trained model as the trained model for primary inference and the trained model for secondary inference. Alternatively, the model generation unit 70 may generate models that are different from each other as the trained model for primary inference and the trained model for secondary inference but have the same model type, network structure, and size as the trained model for primary inference and the trained model for secondary inference.

[0066] Alternatively, the model generation unit 70 may generate models that differ in at least one of the model type, network structure, and size, etc., as the trained model for primary inference and the trained model for secondary inference. For example, when generating both trained models using deep learning, the model generation unit 70 may generate models that differ in at least one of the model type, network structure, and size, as the trained model for primary inference and the trained model for secondary inference. For example, the model generation unit 70 may generate a trained model that is small and lightweight as the trained model for primary inference, compared to the trained model for secondary inference. However, this does not limit the trained models generated by the model generation unit 70.

[0067] The machine learning method used by the model generation unit 70 is arbitrary. For example, the model generation unit 70 may use deep learning to generate at least one of the trained model for primary inference and the trained model for secondary inference. The model generation unit 70 may use the same machine learning method for generating the trained model for primary inference and the machine learning method for generating the trained model for secondary inference, or may use different methods. Note that in the description of the second embodiment, as an example, the model generation unit 70 generates different trained models as the trained model for primary inference and the trained model for secondary inference.

[0068] Then, the model generation unit 70 stores the generated trained model for primary inference and trained model for secondary inference in the model storage unit 80. However, the model generation unit 70 may acquire at least one of the trained model for primary inference and the trained model for secondary inference from an external device not shown. In this case, the model generation unit 70 stores the acquired trained model for primary inference or trained model for secondary inference in the model storage unit 80.

[0069] (8) Model storage unit 80 The model storage unit 80 stores the trained model for primary inference and the trained model for secondary inference. The model storage unit 80 may store a model used by the model generation unit 70 to generate the trained model. For example, an operator may store a model used to generate the trained model in the model storage unit 80 in advance. The information processing device 1 may use an external storage device (not shown) as the model storage unit 80. In this case, the information processing device 1 may not include the model storage unit 80 as a physical configuration. Alternatively, the information processing device 1 may acquire at least one of the trained model for primary inference and the trained model for secondary inference from a device (not shown) and store the acquired model in the model storage unit 80. When acquiring both the trained model for primary inference and the trained model for secondary inference, the information processing device 1 may not include the data storage unit 50, the data generation unit 60, and the model generation unit 70.

[0070] (9) Object reasoning part 10 The object inference unit 10 infers a target object in the inference target data by using a target object in the aggregated data, which is a result of the secondary inference in the secondary inference unit 40, and the aggregated correspondence generated by the data aggregation unit 30 in generating the aggregated data. For example, when the coordinate transformation of BB is stored as the aggregated correspondence, the object inference unit 10 applies the inverse transformation of the coordinate transformation to the coordinates of BB inferred as a result of the secondary inference as the inference position of BB, to calculate the position of BB of the target object in the inference target data. For example, when the rectangle of BB obtained as a result of the secondary inference protrudes from the rectangle in the aggregated data, the object inference unit 10 may discard the BB. Alternatively, for example, when the value obtained by dividing the area of ​​the intersection part of the rectangle of BB obtained as a result of the secondary inference and the rectangle on the aggregated data by the area of ​​the rectangle of BB obtained as a result of the secondary inference is less than a predetermined threshold, the object inference unit 10 may discard the BB. Note that, when the above quotient is equal to or greater than a predetermined threshold, the object inference unit 10 may modify the rectangle of BB to the above intersection part.

[0071] (A) Generation of trained models 2 is a flow diagram showing an example of an operation of generating a trained model in the information processing device 1 according to the second embodiment. The information processing device 1 starts generating a trained model when a predetermined condition is met. For example, the information processing device 1 starts generating a trained model when an instruction from an operator is received. In this case, at the start of the operation, the information processing device 1 may acquire parameters required for generating the trained model from the operator.

[0072] The parameters necessary for generating the trained model are arbitrary. For example, when the data storage unit 50 stores a plurality of training data sets, the parameters are information indicating which training data set is to be used. Alternatively, the parameters may be parameters related to the machine learning process in the model generation unit 70. Note that the information processing device 1 may acquire other information in addition to the parameters. For example, the data generation unit 60 may acquire at least a part of the training data set from the operator. In this case, the data generation unit 60 stores the acquired training data set in the data storage unit 50.

[0073] The data generation unit 60 generates a primary inference learning dataset using the original learning dataset stored in the data storage unit 50 (step S100). For example, the data generation unit 60 generates a primary inference learning dataset using supervised answer data included in the original learning dataset. Then, the data generation unit 60 stores the primary inference learning dataset in the data storage unit 50. The model generation unit 70 performs machine learning in a predetermined model using the primary inference learning dataset to generate a trained model for primary inference (step S101). Then, the model generation unit 70 stores the trained model for primary inference in the model storage unit 80.

[0074] The data generation unit 60 generates a secondary inference learning dataset using the original learning dataset stored in the data storage unit 50 (step S102). For example, the data generation unit 60 may generate aggregated data using the supervised data of the original learning dataset, and generate a secondary inference learning dataset including the supervised data and the aggregated data. Then, the data generation unit 60 stores the generated secondary inference learning dataset in the data storage unit 50. The model generation unit 70 executes machine learning in a predetermined model using the secondary inference learning dataset, and generates a trained model for secondary inference (step S103). Then, the model generation unit 70 stores the trained model for secondary inference in the model storage unit 80. After generating the trained model, the information processing device 1 may notify the operator of the execution result.

[0075] (B) Inference behavior 3 is a flow diagram showing an example of an operation of inferring a target object in the information processing device 1 according to the second embodiment. Note that before the inference operation starts, the model storage unit 80 stores a trained model. However, if the model storage unit 80 does not store a trained model, the information processing device 1 may start the inference operation after performing the operation of "(A) Generation of trained model" to generate a trained model.

[0076] The information processing device 1 starts inference of the target object when a predetermined condition is met. For example, the information processing device 1 starts inference of the target object when an instruction from an operator is met. The information processing device 1 may acquire parameters from the operator in inferring the target object. The parameters are not limited to the following, but may be, for example, a specification of a trained model to be used. However, the information processing device 1 may use parameters given in advance. Alternatively, the information processing device 1 may automatically start inference of the target object after the device is started. Alternatively, the information processing device 1 may execute the operation described below when the data acquisition unit 90 acquires one or a predetermined amount of inference target data. However, the information processing device 1 may start the inference operation asynchronously with the acquisition of the inference target data in the data acquisition unit 90. Note that the data acquisition unit 90 may acquire the inference target data in advance and store it in a storage unit not shown. In this case, the primary inference unit 20 may acquire the inference target data from the storage unit.

[0077] The primary inference unit 20 executes primary inference using the inference target data (step S114). The primary inference unit 20 may collectively execute primary inference on a predetermined amount of inference target data. Hereinafter, the predetermined amount of inference target data on which the primary inference unit 20 collectively executes primary inference is referred to as an "inference target data group." For example, when the inference target data is an image including a target object, the primary inference unit 20 infers, as primary inference, a set of "BB, class, and confidence" pairs from the images of the predetermined amount of inference target data (inference target data group). Furthermore, the primary inference unit 20 may infer sub-target objects in the inference target data group.

[0078] The primary inference unit 20 may determine the inference target data group to be inferred collectively using a condition other than the quantity of the inference target data. For example, the primary inference unit 20 may determine the inference target data group to be inferred collectively using at least one of the following conditions: (a) When the number of BBs inferred in the first inference reaches a predetermined number, (b) When the sum of the sizes of BBs inferred in the first inference reaches a predetermined value, (c) inferring the number of aggregated data, and when the inferred number of aggregated data reaches a predetermined value, (d) Inference target data for which a predetermined time has passed since it was acquired by the information processing device 1. However, the primary inference unit 20 may execute primary inference for all of the acquired inference target data. When executing primary inference for an inference target data group, the primary inference unit 20 may process the primary inference for at least a part of the inference target data included in the inference target data group in parallel, sequentially, or by combining parallel and sequential processing.

[0079] Fig. 5 is a diagram showing an example of a result of the primary inference. In Fig. 5, of the two rectangles, the smaller inner rectangle is the BB of the LP, which is the target object obtained as a result of the primary inference. The larger outer rectangle is the BB of the front and rear surfaces of the car, which are sub-target objects obtained as a result of the primary inference. Note that Fig. 5 omits the display of classes and confidences. Returning to the explanation referring to Fig. 3.

[0080] The data aggregating unit 30 performs post-processing on some or all of the results of the primary inference (step S115). For example, the data aggregating unit 30 performs filtering and adjustment of the BB using the attributes of the BB and information on the positional relationship between the target object and the sub-target object. FIG. 4 is a flow diagram showing an example of the post-processing in the data aggregating unit 30. More specifically, FIG. 4 shows an example of the operation of the post-processing in the data aggregating unit 30 when the BB is inferred as a result of the primary inference. Hereinafter, with reference to FIG. 4, the operation when the target object is the LP and the sub-target object is the front and rear surfaces of the vehicle including the LP will be described.

[0081] The data aggregating unit 30 judges whether or not the BB is a BB of the target object (step S131). The case where the BB is not a BB of the target object means the case where the BB is a BB of a sub-target object. If the BB is not a BB of the target object (No in step S131), the data aggregating unit 30 judges whether or not the sub-target object has a predetermined relationship with the target object included in the result of the primary inference (step S132). For example, the data aggregating unit 30 judges whether or not the BB of the sub-target object includes the BB of the target object included in the result of the primary inference. If the BB is a BB of the target object (Yes in step S131) ​​or if the sub-target object does not have a predetermined relationship (No in step S132), the data aggregating unit 30 judges whether or not the BB has a shape appropriate for use as aggregated data (step S133). For example, the data aggregating unit 30 judges whether or not the size and aspect ratio of the BB are within a predetermined threshold range.

[0082] If the BB has an appropriate shape (Yes in step S133), the data aggregating unit 30 modifies the BB to a shape appropriate for generating aggregated data (step S134). For example, the data aggregating unit 30 sets the size and aspect ratio of the BB to values ​​appropriate for generating aggregated data. For example, the data aggregating unit 30 may change the size of the BB by multiplying the height and width of the BB by a predetermined value. Alternatively, the data aggregating unit 30 may modify the height and width of the BB so that the aspect ratio of the BB falls within a predetermined range. If the sub-target object has a predetermined relationship (Yes in step S132) or if the BB does not have an appropriate shape (No in step S133), the data aggregating unit 30 discards the BB (step S135). That is, the data aggregating unit 30 does not include the BB in the aggregated data. In this way, the data aggregating unit 30 performs post-processing on the result of the primary inference. Returning to the description with reference to FIG. 3.

[0083] The data aggregating unit 30 generates aggregated data using all the results of the primary inference after post-processing (step S116). For example, when the inference target data is an image, the data aggregating unit 30 generates an image that collects images of the regions of the target object and the regions of the sub-target object inferred as the result of the primary inference. Note that the data aggregating unit 30 may generate aggregated data using an area that combines at least a part of overlapping areas and adjacent areas in the result of the primary inference.

[0084] FIG. 6 is a diagram showing an example of aggregated data. In FIG. 6, the display of classes and confidences is omitted. The aggregated data in FIG. 6 is an example of aggregated data generated using a total of five data, including the data in FIG. 5 and four data before and after FIG. 5 in terms of time. Furthermore, the aggregated data in FIG. 6 is an example of a case where an LP is detected in all of the data in the first inference for those five data. In addition, if an LP is not detected in any of the data and the front and rear surfaces of the vehicle are detected, the image corresponding to that data is an image of the front and rear surfaces of the vehicle rather than the LP. In the example of FIG. 6, the data aggregation unit 30 makes the area of ​​the image to be extracted larger than the area of ​​the BB of the result of the first inference in order to reduce the possibility that a part of the target object is missing. For example, the image of each LP is an image of an area larger than the BB of the LP. In the example shown in FIG. 6, the total area of ​​the images as a result of the first inference is smaller than the area of ​​the image of the aggregated data. Therefore, the image of the aggregated data in FIG. 6 includes a surplus area. The black area on the right side of FIG. 6 is the surplus area. For example, when further aggregating the results of the primary inference, the data aggregating unit 30 may add at least a part of the results of the primary inference to be aggregated to the surplus area. If the results of the primary inference to be added do not fit in the surplus area, the data aggregating unit 30 may generate new aggregated data. Note that the sub-target object in FIG. 5 includes a target object included in the result of the primary inference inside. In other words, the sub-target object in FIG. 5 is a sub-target object that has a predetermined positional relationship with the target object. Therefore, the BB of the sub-target object in FIG. 5 is not included in the aggregated data in FIG. 6. Returning to the explanation with reference to FIG. 3.

[0085] Furthermore, in generating the aggregated data, the data aggregating unit 30 generates an aggregated correspondence relationship between the positions of the data areas of the target object and sub-target objects included in the inference target data and the positions of the data areas of the target object and sub-target objects in the aggregated data. The secondary inference unit 40 executes secondary inference using the aggregated data (step S118). FIG. 7 is a diagram showing an example of the result of secondary inference. The rectangle in FIG. 7 is the BB of the LP of the target object inferred by the secondary inference unit 40. Note that FIG. 7 omits the display of the class and confidence. Returning to the explanation referring to FIG. 3.

[0086] The object inference unit 10 infers a target object in the inference target data using the result of the secondary inference and the aggregated correspondence relationship (step S119). FIG. 8 is a diagram showing an example of inference of a target object in the inference target data. Compared with FIG. 5, the position of BB in FIG. 8 is an appropriate position with respect to LP. Note that, if the size of a target object included in the result of the secondary inference is larger than the size of the data aggregated in the aggregated data, the object inference unit 10 may ignore the target object. Returning to the explanation with reference to FIG. 3.

[0087] The information processing device 1 repeats the above operations until a predetermined condition is satisfied. For example, the information processing device 1 may end the operations when it receives an end instruction from an operator. Alternatively, the information processing device 1 may repeat the above operations until there is no inference target data to be processed, until the amount of processed inference target data reaches a predetermined value, until a predetermined time has elapsed, or until the number of repetitions reaches a predetermined number. Based on the above operations, the information processing device 1 infers a target object in the inference target data.

[0088] As explained so far, aggregated data is data that collects areas of target objects in inference target data, and is data of smaller quantity than the inference target data. Therefore, the load of secondary inference is lower than the load when all of the inference target data is used. Furthermore, primary inference only needs to be able to infer the target object with a certain degree of accuracy. In other words, the information processing device 1 can use inference with a lower load than secondary inference as primary inference. As a result, the information processing device 1 can improve the throughput in inference.

[0089] Using a specific example, the improvement of the inference throughput in the information processing device 1 will be described. As an inference to be compared with the information processing device 1, an inference in which a model equivalent to the trained model for secondary inference is applied to the inference target data is used. Hereinafter, an inference in which an equivalent model is applied is referred to as "associated inference." Note that in the following description, the following is assumed as a premise. (1) The data to be inferred is an image. (2) The performance and scale of the association inference are the same as those of the secondary inference in the secondary inference unit 40. The time required for the association inference and secondary inference to infer one image is the unit time. In the following, the unit time is defined as "1". (3) The time required for the primary inference in the primary inference unit 20 to infer one image is one-tenth (0.1) of that required for the association inference and secondary inference. (4) On average, the data aggregating unit 30 generates one aggregated image from three images to be inferred. In other words, the ratio of the number of images to be inferred to the number of images of the aggregation result is 3:1. (5) The total number of images is N. (6) Processing other than inference is ignored because its processing time is small compared to inference.

[0090] In this case, the inference time of the association inference is "N×1=N". The inference time of the primary inference in the information processing device 1 is "N×0.1=0.1N". The inference time of the secondary inference is "N / 3×1=N / 3". In other words, the inference time of the information processing device 1 is "approximately 0.43N (=0.1N+N / 3)". In this way, the information processing device 1 can make inference in a shorter time than the association inference. As a result, the information processing device 1 can improve the throughput compared to the association inference. In addition, the scale of the trained model for secondary inference used for the secondary inference, which is the final inference in the information processing device 1, is equivalent to the model used for the association inference. Therefore, the accuracy of the inference of the information processing device 1 is equivalent to that of the association inference.

[0091] In addition, a lightweight model tends to be faster but less accurate than a heavy model. For example, in the above example, the accuracy of the primary inference in the primary inference unit 20 tends to be lower than the accuracy of the secondary inference in the secondary inference unit 40. Therefore, the primary inference unit 20 may infer a second target object that is in a predetermined positional relationship with the first target object. In this case, the data aggregation unit 30 may generate aggregated data and a correspondence relationship using the first target object and the second target object. Using the above operation, the information processing device 1 can perform inference using not only the target object of the inference in the information processing device 1 but also a sub-target object that is in a predetermined relationship with the target object. Therefore, the information processing device 1 can improve the accuracy of inference. For example, the information processing device 1 can suppress overlooking of the target object.

[0092] Furthermore, the data aggregating unit 30 may use a second target object that is not included in the result of the primary inference, with respect to the first target object that is in a predetermined positional relationship with the second target object included in the result of the primary inference. Using the above operation, the information processing device 1 can reduce the size of the aggregated data. As a result, the information processing device 1 can improve the throughput. Furthermore, the data aggregating unit 30 may perform a predetermined process, such as post-processing, on at least a part of the first target object included in the result of the primary inference before generating the aggregated data. Using the above operation, the information processing device 1 can generate more appropriate aggregated data. As a result, the information processing device 1 improves the accuracy of the inference. For example, the information processing device 1 suppresses overlooking of the target object. Note that, when using a sub-target object, the information processing device 1 may perform a predetermined process on at least a part of the second target object included in the result of the primary inference before generating the aggregated data.

[0093] The data aggregating unit 30 corrects the size of the BB by adding or multiplying at least one of the height and width of the BB of the target object and the BB of the sub-target object by a predetermined value. However, the second embodiment is not limited to this. As another example, the data aggregating unit 30 may correct the size of the BB by multiplying at least one of the height and width of the BB of the target object and the BB of the sub-target object by a predetermined value and then adding a predetermined value. Alternatively, the data aggregating unit 30 may correct the size of the BB by applying a predetermined linear or nonlinear transformation to at least one of the height and width of the BB of the target object and the BB of the sub-target object. In this case, the parameters related to the transformation may be given to the data aggregating unit 30 in advance, for example. Alternatively, the data aggregating unit 30 may switch the transformation method and the parameters related to the transformation based on the attributes of the BB and the inference target data, or other information. The other information is, but is not limited to, for example, the load of the information processing device 1.

[0094] In the above description, an example has been given in which there is one primary inference unit 20 and one secondary inference unit 40. However, the configuration of the information processing device 1 is not limited to this. For example, the information processing device 1 may include a plurality of primary inference units 20. In this case, the plurality of primary inference units 20 may each perform inference using different parameters. The parameters are, for example, a trained model, a data size, or a recognition target class. Furthermore, all of the plurality of primary inference units 20 may perform inference using the same inference target data. Alternatively, at least some of the primary inference units 20 may perform inference using inference target data different from that of the other primary inference units 20.

[0095] Alternatively, for example, the information processing device 1 may include two or more secondary inference units 40. When there are multiple secondary inference units 40, at least some of the secondary inference units 40 may perform inference using parameters different from those of the other secondary inference units 40. The parameters in this case are, for example, a trained model or a data size. When there are multiple secondary inference units 40, the data aggregating unit 30 may generate aggregated data corresponding to each secondary inference unit 40. When generating aggregated data corresponding to each of the multiple secondary inference units 40, the data aggregating unit 30 may generate aggregated data using different parameters for each secondary inference unit 40.

[0096] Alternatively, the data aggregating unit 30 may divide the result of the primary inference to generate aggregated data, or may divide the generated aggregated data and allocate it to multiple secondary inference units 40. When allocating to multiple secondary inference units 40, the data aggregating unit 30 may generate or divide the aggregated data using, for example, the following attribute of BB in the result of the primary inference or a combination of these attributes. Note that the data aggregating unit 30 allocates aggregated data to multiple secondary inference units 40 when the data aggregating unit 30 generates aggregated data corresponding to each of the multiple secondary inference units 40, or when the data aggregating unit 30 divides the aggregated data generated by the data aggregating unit 30. (1) BB size or aspect ratio, (2) BB's class, (3) BB confidence, (4) BB's position in the results of the first-order inference, (5) If BB is the BB of the target object, whether the corresponding sub-target object is detected or not.

[0097] The data aggregation unit 30 may include a part of the result of the primary inference in the multiple aggregated data to be generated or in the multiple aggregated data after division. The data aggregation unit 30 may use the above-mentioned attributes or a combination thereof to determine whether or not to include the result of the primary inference in the multiple aggregated data. When the result of the primary inference is included in the multiple aggregated data, the multiple secondary inference units 40 may infer the same target object. In this case, the object inference unit 10 may determine the overlap of the target object using the result of the secondary inference and the aggregated correspondence relationship, and execute an operation corresponding to the overlap using the determination result. For example, the object inference unit 10 may correct the result of the inference using the overlap. For example, the object inference unit 10 may select, as the BB of the target object, a BB with the highest confidence among the multiple BBs for the target object that are inferred in an overlapping manner. Alternatively, the object inference unit 10 may average the rectangular information of the BB in the multiple inference results as the rectangular information of the BB of the target object. For the averaging, the object inference unit 10 may use a weighted average based on confidence.

[0098] The accuracy of the inference result depends on the inference difficulty of the input data. If the data has a low inference difficulty, even a trained model with a low inference accuracy can often perform inference with high accuracy. A trained model with a low inference accuracy is, for example, a lightweight, high-speed, or low processing load model. On the other hand, if the data has a high inference difficulty, it is difficult to achieve high inference accuracy when a trained model with a low inference accuracy is used. In order to achieve sufficient inference accuracy, it is necessary to use an inference model with high inference accuracy. An inference model with high inference accuracy is a model with a high processing load. Therefore, by using different trained models for inference according to the characteristics of the data to be inferred, such as the inference difficulty, it is expected that both inference speed and inference accuracy can be achieved. The same can be said for inference parameters such as the size of a convolutional neural network (CNN).

[0099] Therefore, the information processing device 1 may classify the inference target data or the results of the primary inference into a plurality of groups based on characteristics such as the inference difficulty level, and perform secondary inference on each of the classified groups. Such a case will be described below. In the following description, a posture estimation task using keypoints is used as the task to be executed. However, this does not limit the present embodiment. For example, in the following description, the present embodiment may be used for other tasks such as an object detection task instead of the posture estimation task. In addition, the characteristics used for classification are not limited to the inference difficulty level, and may be other characteristics.

[0100] 14 is a block diagram showing an example of the configuration of an information processing device 1D that executes secondary inference for each group. The information processing device 1D includes an object inference unit 10D, a primary inference unit 20D, a data aggregation unit 30D, a plurality of secondary inference units 40D, a data storage unit 50, a data generation unit 60, a model generation unit 70, a model storage unit 80, and a data acquisition unit 90. Note that the data storage unit 50, the data generation unit 60, the model generation unit 70, the model storage unit 80, and the data acquisition unit 90 are the same as those of the information processing device 1 except that there are a plurality of secondary inference units 40D, and therefore detailed explanations thereof will be omitted.

[0101] Like the object inference unit 10, the object inference unit 10D infers the posture of the target object in the inference target data using the result of the secondary inference in the secondary inference unit 40D and the aggregated correspondence generated by the data aggregation unit 30D in generating the aggregated data. More specifically, in this case, the object inference unit 10D infers the posture of the target object based on the result of the keypoint extraction process of the posture estimation task, which is the result of the secondary inference in the secondary inference unit 40D. For example, when the coordinate transformation of BB is acquired as the aggregated correspondence, the object inference unit 10D applies the inverse transformation of the coordinate transformation to the coordinates of each keypoint inferred as a result of the secondary inference, to calculate the coordinates of the keypoint in the inference target data.

[0102] The primary inference unit 20D executes primary inference in the same way as the primary inference unit 20. However, in this case, as an example, the primary inference unit 20D executes a keypoint extraction process as a posture detection task. For example, the primary inference unit 20D outputs the coordinates, class, and confidence of each keypoint included in the inference target data as an inference result. The class is, for example, eyes or nose, but is not limited to this. Furthermore, the primary inference unit 20D may execute a process of generating a posture as a posture detection task.

[0103] The secondary inference unit 40D executes secondary inference in the same way as the secondary inference unit 40. For example, in this case, the secondary inference unit 40D executes a keypoint extraction process as a posture detection task in the same way as the primary inference unit 20D. The secondary inference unit 40D outputs, for example, the coordinates, class, and confidence of each keypoint included in the inference target data as an inference result. The class is, for example, eyes or nose, but is not limited to this. Furthermore, the secondary inference unit 40D may execute a process of generating a posture as a posture detection task. However, the secondary inference units 40D execute secondary inference on aggregated data of different groups.

[0104] The secondary inference units 40D may use the same trained model or may use different trained models. The same applies to the parameters of inference. In the following description, the information processing device 1D includes two secondary inference units 40D as an example. When the two secondary inference units 40D are to be distinguished from each other, they are called the secondary inference unit 40D-1 and the secondary inference unit 40D-2. For convenience of explanation, in the following description, the secondary inference unit 40D-1 uses a model with higher accuracy than the model used by the secondary inference unit 40D-2. In other words, the secondary inference unit 40D-2 uses a model with lower accuracy but higher speed than the secondary inference unit 40D-1. The secondary inference units 40D use aggregated data of different groups. Therefore, the information processing device 1D may include any number of secondary inference units 40D as long as the number of aggregated data generated by the data aggregation unit 30D is greater than the number of groups. For example, when the number of secondary inference units 40D is greater than the number of groups, some of the secondary inference units 40D may perform secondary inference using different trained models for the same aggregated data.

[0105] The data aggregating unit 30D classifies the first target object inferred in the primary inference into one of the groups based on the attributes of the first target object. Then, the data aggregating unit 30D generates aggregated data for each group, which is data with a smaller amount than the inference target data. Then, for each aggregated data, the data aggregating unit 30D generates a correspondence relationship between the position of the first target object in the inference target data and the position of the first target object in the aggregated data.

[0106] For example, the data aggregating unit 30D estimates the BB as an attribute of the first target object based on the result of the keypoint extraction process of the primary inference. An example of BB estimation in the data aggregating unit 30D will be described. The data aggregating unit 30D regards the coordinates obtained as a result of the primary inference for each keypoint as a 1×1 rectangle, and estimates the BB corresponding to the keypoint. However, the data aggregating unit 30D may estimate a BB different from this. Furthermore, the data aggregating unit 30D may combine overlapping or adjacent BBs into one BB. The data aggregating unit 30D may use, for example, a comparison between a predetermined threshold and the distance between the BBs to determine whether or not they are adjacent. The data aggregating unit 30D may perform these processes for each class of the result of the primary inference, or across multiple classes. The data aggregating unit 30D may use different thresholds as the thresholds used for the determination for each class. For example, the data aggregating unit 30D may use different thresholds corresponding to the result of the primary inference or the attribute of the BB. The attribute of a BB is, for example, the size of the BB, the position of the BB, the number of keypoints included in the BB, or the class of the BB.

[0107] Then, the data aggregating unit 30D classifies the results of the primary inference into groups using the estimated BB, and executes "data aggregation" including "post-processing" and "aggregation" for each group. The operation of data aggregation is the same as that of the data aggregating unit 30. For example, in this case, the data aggregating unit 30D classifies the results of the primary inference into two groups, one for the secondary inference unit 40D-1 and the other for the secondary inference unit 40D-2, using information on the attributes of the estimated BB. Note that the data aggregating unit 30D may perform classification before post-processing, during post-processing, or after post-processing. That is, the data aggregating unit 30D may perform post-processing and classification in any order, or in parallel. Then, the data aggregating unit 30D generates aggregated data for the results of the primary inference classified into the same group. The data aggregating unit 30D may classify the target objects of all classes as the objects to be classified into groups, or may classify the target objects of some classes and sub-target objects of the target objects of another class.

[0108] For example, the data aggregating unit 30D may operate as follows. The accuracy of the model processing of the secondary inference unit 40D-1 is higher than that of the secondary inference unit 40D-2. Therefore, the data aggregating unit 30D may estimate the inference difficulty of each BB in the secondary inference and classify them into groups based on the result. For example, the data aggregating unit 30D classifies BBs with high inference difficulty into the group of the secondary inference unit 40D-1, and BBs with low inference difficulty into the group of the secondary inference unit 40D-2. However, the classification of the data aggregating unit 30D is not limited to this. For example, the data aggregating unit 30D may classify based on the size of the BB or based on the confidence of the BB. For example, the data aggregating unit 30D may classify BBs with small size or low confidence into the group of the secondary inference unit 40D-1, and BBs with large size or high confidence into the group of the secondary inference unit 40D-2. However, the data aggregating unit 30D may execute classification based on machine learning including deep learning, not limited to the rule-based classification as described above.

[0109] Next, an example of estimation of the inference difficulty in the secondary inference in the data aggregating unit 30D will be described. For example, the data aggregating unit 30D may estimate that a BB including only an upright posture has a low inference difficulty, and may estimate that a BB that may include postures other than upright has a high inference difficulty. As an estimation determination, for example, the data aggregating unit 30D may estimate that the posture is upright when the aspect ratio of the rectangle of the BB or the circumscribing rectangle of the keypoint group included in the BB is within a predetermined range. Furthermore, the data aggregating unit 30D may estimate the number of postures in the BB, and when there is one posture, may perform estimation based on the aspect ratio. In this case, the data aggregating unit 30D may use, for example, the maximum value of the number of keypoints included in the BB tallied for each class as an estimate of the number of postures. Alternatively, the data aggregating unit 30D may use a value excluding the maximum and minimum values ​​of the keypoints, or a value subjected to statistical processing such as an average value as an estimate. The data aggregating unit 30D may estimate the number of postures by prioritizing the classes. Data aggregator 30D may further estimate the number of poses using the confidence of the keypoint. Alternatively, when object inference unit 10D executes a pose generation process of the pose estimation task, data aggregator 30D may obtain a result of the pose generation process and estimate the number of poses using the result.

[0110] Alternatively, the data aggregating unit 30D may estimate the inference difficulty to be low when the size or area of ​​the BB is larger than a predetermined value. Alternatively, the data aggregating unit 30D may estimate the number of postures in the BB, and estimate the inference difficulty to be low when the estimated number of postures or the value obtained by dividing the size of the BB by the square of the estimated number of postures is larger than a predetermined value. When using this estimation, the data aggregating unit 30D can avoid estimating the inference difficulty of a BB that includes a small number of postures but has a large size to be low.

[0111] In changing the size of the target object and the sub-target object at the time of aggregation, the data aggregation unit 30D may determine at least one of the following: whether to apply resizing, the resizing method, and the resizing parameters, using the result of classification and information generated during the classification process. Note that the information generated during the classification process is, for example, the inference difficulty in the secondary inference, whether the posture is upright, the number of estimated postures, or the height and width of the BB.

[0112] Next, the operation of the information processing device 1D will be described with reference to the drawings. FIG. 15 is a flow diagram showing an example of the operation of inferring a target object in the information processing device 1D of FIG. 14. The primary inference unit 20D executes primary inference using inference target data (step S114D) in the same manner as step S114 in FIG. 3. However, in this case, the primary inference unit 20D executes inference processing of a posture estimation task. The data aggregating unit 30D executes post-processing on some or all of the results of the primary inference in the same manner as step S115 in FIG. 3. However, in this case, the data aggregating unit 30D estimates BB from the results of the primary inference in addition to the operation of step S115. The data aggregating unit 30D classifies the results of the primary inference after the post-processing (step S401D). For example, the data aggregating unit 30D estimates the inference difficulty in the secondary inference for the results of the primary inference after the post-processing, and classifies the results of the primary inference after the post-processing based on the inference difficulty. For example, the data aggregating unit 30D classifies the results of primary inferences with high inference difficulty into a group for the secondary inference unit 40D-1, and classifies the results of primary inferences with low inference difficulty into a group for the secondary inference unit 40D-2. The data aggregating unit 30D may perform classification before or during post-processing.

[0113] Then, the data aggregating unit 30D generates aggregated data using all the results of the primary inference after the post-processing, similar to step S116 in FIG. 3 (step S116D). However, the data aggregating unit 30D generates aggregated data for each classified group. For example, the data aggregating unit 30D generates aggregated data for the secondary inference unit 40D-1 and aggregated data for the secondary inference unit 40D-2 based on the classification result of step S401D. The secondary inference unit 40D-1 and the secondary inference unit 40D-2 each perform secondary inference using the aggregated data, similar to step S118 in FIG. 3 (step S118D). In this case, the secondary inference unit 40D-1 and the secondary inference unit 40D-2 perform inference processing of the posture estimation task using aggregated data of different groups. Then, the object inference unit 10D infers the target object in the inference target data, using the result of the secondary inference and the aggregated correspondence, similar to step S119 in FIG. 3 (step S119D). However, in this case, object inference unit 10D executes the pose estimation task using the inference results of secondary inference unit 40D-1 and secondary inference unit 40D-2.

[0114] As described above, the information processing device 1D includes an object inference unit 10D, a primary inference unit 20D, a data aggregation unit 30D, and a plurality of secondary inference units 40D. The primary inference unit 20D infers the first target object by applying inference target data, at least a portion of which includes the first target object, to a first learned model as a primary inference. The data aggregation unit 30D classifies the first target object inferred in the primary inference into one of the groups based on the attributes of the first target object. Then, the data aggregation unit 30D generates aggregated data, which is data with a smaller amount than the inference target data for each group. Furthermore, the data aggregation unit 30D generates an aggregated correspondence relationship, which is a correspondence relationship between the position of the first target object in the inference target data and the position of the first target object in the aggregated data. The secondary inference unit 40D infers the first target object by applying aggregated data of different groups to a second learned model as a secondary inference. The object inference unit 10D infers the first target object in the inference target data, using the first target object in the result of the secondary inference and the aggregate correspondence relationship, which is the correspondence relationship.

[0115] In this way, the information processing device 1D includes a plurality of secondary inference units 40D. Then, the data aggregation unit 30D classifies the results of the primary inference into groups based on the attributes of the first target object, such as the inference difficulty in the secondary inference. Then, the data aggregation unit 30D generates aggregated data for each group. Then, the plurality of secondary inference units 40D execute secondary inference using the aggregated data for each group. Therefore, for example, for a group with low inference difficulty, the information processing device 1D can improve the inference speed while maintaining accuracy by using the secondary inference unit 40D that uses a learned model or inference parameter that is not highly accurate but is lightweight and fast as secondary inference. On the other hand, for a group with high inference difficulty, the information processing device 1D can execute inference that avoids deterioration of accuracy by using the secondary inference unit 40D that uses a learned model or inference parameter that is highly accurate. In this way, the information processing device 1D can improve the throughput of inference of the target object while maintaining accuracy. In this way, the secondary inference units 40D may use different learned models or different inference parameters. Furthermore, the data aggregating unit 30D may estimate the inference difficulty in each of the secondary inference units 40D, and classify the first target object into one of the groups based on the inference difficulty.

[0116] [Variations] In the above description, an example of using the inference of the posture estimation task as the primary inference and the secondary inference has been shown. However, this embodiment is not limited to the posture estimation task, and may use, for example, an object detection task. In addition, in the above description, the data aggregation unit 30D generates information used for classification, such as the inference difficulty, but this embodiment is not limited to this. For example, the primary inference unit 20D may determine the inference difficulty in addition to the result of the primary inference and output it to the data aggregation unit 30D. That is, the primary inference unit 20D may infer the inference difficulty in each of the secondary inference units 40D. For example, the primary inference unit 20D may infer the inference difficulty for each keypoint in the posture estimation task or for each BB in the object detection task in the learned model of the primary inference. In this case, the primary inference unit 20D may use a class corresponding to the inference difficulty as a class to be output as a result of the primary inference.

[0117] Also, the data aggregating unit 30D may classify at least some of the results of the primary inference into a plurality of groups. For example, the data aggregating unit 30D may include the results of the primary inference with a very high inference difficulty in a plurality of aggregated data. In this case, at least some of the plurality of secondary inference units 40D estimate the same target object included in each of the plurality of aggregated data. Then, the object inference unit 10D integrates the inference results of the plurality of secondary inference units 40D to infer the target object. Using such an operation, the information processing device 1D can improve the recognition accuracy of the target object. However, in this case, there is a possibility that a plurality of inference results are obtained for one object. Therefore, the object inference unit 10D may correct the overlapping inference results using a technique such as Non-Maximum Suppression (NMS).

[0118] [Hardware configuration] Next, the hardware of the information processing device 1 and 1D will be described using the information processing device 1D. In the above description, an example in which the object inference unit 10D, the primary inference unit 20D, the data aggregation unit 30D, the secondary inference unit 40D, the data storage unit 50, the data generation unit 60, the model generation unit 70, the model storage unit 80, and the data acquisition unit 90 are included in one device has been described. However, the second embodiment is not limited to this. For example, the information processing device 1D may be configured by connecting devices having functions corresponding to each configuration via a predetermined network. For example, the information processing device 1D may be realized using cloud computing. Alternatively, each component of the information processing device 1D may be configured by a hardware circuit. Alternatively, in the information processing device 1D, multiple components may be configured by one hardware.

[0119] Alternatively, the information processing device 1D may be realized as a computer device including a CPU, a read only memory (ROM), and a random access memory (RAM). The information processing device 1D may be realized as a computer device including a network interface circuit (NIC) in addition to the above configuration. Alternatively, the information processing device 1D may be realized as a computer device including an arithmetic unit (Arithmetic Logical Unit (ALU)) that executes some or all of the operations of learning and inference. FIG. 9 is a block diagram showing the configuration of a computer device 600, which is an example of the hardware configuration of the information processing devices 1 and 1D. The computer device 600 in FIG. 9 includes a CPU 610, an ALU 611, a ROM 620, a RAM 630, a storage device 640, and a NIC 650, and constitutes a computer device.

[0120] The CPU 610 reads a program from at least one of the ROM 620 and the storage device 640. Then, the CPU 610 controls the ALU 611, the RAM 630, the storage device 640, and the NIC 650 based on the read program. The CPU 610 then realizes the functions of an object inference unit 10D, a primary inference unit 20D, a data aggregation unit 30D, a secondary inference unit 40D, a data storage unit 50, a data generation unit 60, a model generation unit 70, a model storage unit 80, and a data acquisition unit 90.

[0121] When implementing each function, the CPU 610 may use the RAM 630 or the storage device 640 as a temporary storage medium for the program. The CPU 610 may also read, using a storage medium reading device (not shown), a program contained in a storage medium 690 that stores a computer-readable program. Alternatively, the CPU 610 may receive a program from an external device (not shown) via the NIC 650, store the program in the RAM 630 or the storage device 640, and operate based on the stored program.

[0122] The ALU 611 is responsible for a predetermined arithmetic processing and a part of a predetermined processing in the CPU 610. For example, the ALU 611 is controlled by the CPU 610 to execute a part or all of the arithmetic operations of learning and inference. The configuration of the ALU 611 is arbitrary. For example, the ALU 611 may be a graphics processing unit (GPU) or a field programmable gate array (FPGA). Alternatively, the ALU 611 may be, for example, an application specific integrated circuit (ASIC). Note that information (data, programs, circuit information, etc.) required for the calculation in the ALU 611 is stored in advance in the ROM 620, the RAM 630, or the storage device 640.

[0123] The ROM 620 stores programs and fixed data executed by the CPU 610. The ROM 620 is, for example, a P-ROM (Programmable-ROM) or a flash ROM. The RAM 630 temporarily stores programs and data executed by the CPU 610. The RAM 630 is, for example, a D-RAM (Dynamic-RAM). The storage device 640 stores data and programs that the computer device 600 stores for a long period of time. The storage device 640 may operate as a data storage unit 50. The storage device 640 may also operate as a model storage unit 80. The storage device 640 may also operate as a temporary storage device for the CPU 610. The storage device 640 may also operate as a secondary storage device for the memory (for example, the ROM 620 and the RAM 630). The storage device 640 is, for example, a hard disk device, a magneto-optical disk device, a solid state drive (SSD), or a disk array device.

[0124] The ROM 620 and the storage device 640 are non-volatile (non-transitory) recording media. On the other hand, the RAM 630 is a volatile (transitory) recording medium. The CPU 610 can operate based on a program stored in the ROM 620, the storage device 640, or the RAM 630. In other words, the CPU 610 can operate using a non-volatile recording medium or a volatile recording medium.

[0125] The NIC 650 relays data exchange with external devices (not shown) via a network. The NIC 650 operates as a part of the data acquisition unit 90 and the object inference unit 10D. The NIC 650 is, for example, a local area network (LAN) card. The NIC 650 is not limited to being wired, and may be wireless.

[0126] The computer device 600 in Fig. 9 configured in this manner can obtain the same effect as the information processing device 1 in Fig. 1 and the information processing device 1D in Fig. 14. This is because the CPU 610 of the computer device 600 in Fig. 9 can realize functions similar to those of the information processing device 1 and the information processing device 1D based on a program. Alternatively, the CPU 610 and ALU 611 of the computer device 600 in Fig. 9 can realize functions similar to those of the information processing device 1 and the information processing device 1D based on a program. Note that the information processing device 1C shown in Fig. 13 may be configured using the computer device 600 in Fig. 9.

[0127] [system] An example of a system including the information processing device 1, 1C, or 1D will be described using the information processing device 1D. FIG. 10 is a block diagram showing an example of a configuration of an information processing system 400 including the information processing device 1D. The information processing system 400 includes the information processing device 1D, a data acquisition device 200, and a display device 300. The information processing system 400 may include multiple devices as each device. For example, the information processing system 400 may include multiple data acquisition devices 200. Note that the information processing system 400 may include a device not shown. For example, the information processing system 400 may include a device that recognizes predetermined information related to a target object using an inference result of the information processing device 1D. The device that recognizes predetermined information related to a target object is, for example, an OCR device.

[0128] The data acquisition device 200 outputs the inference target data to the information processing device 1D. The data acquisition device 200 may output an original learning data set for generating a learning data set to the information processing device 1D. The data acquisition device 200 is, for example, a surveillance camera. In this case, the data acquisition device 200 outputs a captured image to the information processing device 1D as the inference target data. The information processing device 1D operates as described above. That is, the information processing device 1D acquires the inference target data from the data acquisition device 200. Then, the information processing device 1D executes inference on the acquired inference target data and outputs the inference result to the display device 300. The display device 300 acquires the inference result from the information processing device 1D and displays the acquired inference result. The display device 300 is, for example, a liquid crystal display, an organic electroluminescence display, or electronic paper. Specifically, the display device 300 is, for example, a liquid crystal display of a surveillance system. The operator can check the target object by referring to the display of the display device 300.

[0129] Thus, the information processing system 400 includes the information processing device 1D, the data acquisition device 200, and the display device 300. The information processing device 1D operates as described above. The data acquisition device 200 outputs inference target data to the information processing device 1D. The display device 300 acquires an inference result from the information processing device 1D and displays the acquired inference result. Using the above configuration, the information processing system 400 improves inference throughput and displays the target object inferred in the inference target data to an operator or the like.

[0130] <Third embodiment> The load of the primary inference and the secondary inference differs depending on the trained model used for inference. For example, a trained model with high inference accuracy generally has a high load. However, when a trained model with a high load is used, the throughput decreases. Alternatively, the load of data aggregation differs depending on the post-processing and aggregation processing in data aggregation. Alternatively, the method of generating aggregated data and the format of aggregated data used in data aggregation affect the inference accuracy. Note that the method of generating aggregated data used in data aggregation is, for example, how to group regions. Alternatively, the format of aggregated data is, for example, the size and the gap between images.

[0131] In this way, a change in at least one of the primary inference, secondary inference, and data aggregation affects the throughput and inference accuracy in the information processing devices 1, 1C, and 1D. Hereinafter, the primary inference, secondary inference, and data aggregation are collectively referred to as "inference parameters." In general, there is a trade-off between inference accuracy and throughput. It should be noted that there is an inverse proportional relationship between the processing load and the throughput. For example, increasing the processing load decreases the throughput, and decreasing the processing load increases the throughput. Therefore, in order to prevent a decrease in inference accuracy while ensuring a desired throughput, it is desirable to select and use an appropriate inference parameter based on the load or throughput.

[0132] Therefore, as the third embodiment, an embodiment in which at least one of the inference parameters is changed based on the load or throughput will be described. Hereinafter, the third embodiment will be described with reference to the drawings. In the description of the third embodiment, the same reference numerals are used for the configurations and operations as those of the second embodiment, and detailed description thereof may be omitted. In the following description, the third embodiment changes the trained model as the change of the inference parameters. However, this does not limit the third embodiment. In the third embodiment, an inference parameter different from the trained model may be changed according to the load or throughput. The change of the inference parameter will be described later. For convenience of description, the information processing device 1B used in the following description includes one secondary inference unit 40B as the secondary inference unit. When multiple secondary inference units 40B are used as in the information processing device 1D, in the following description, each configuration of the information processing device 1B may include a function similar to that of the corresponding configuration in the information processing device 1D.

[0133] Fig. 11 is a block diagram showing an example of the configuration of an information processing device 1B according to the third embodiment. The information processing device 1B includes an object inference unit 10, a primary inference unit 20B, a data aggregation unit 30B, a secondary inference unit 40B, a data storage unit 50B, a data generation unit 60B, a model generation unit 70B, a model storage unit 80B, and a data acquisition unit 90. The information processing device 1B may be configured using the computer device 600 of Fig. 9, similarly to the second embodiment.

[0134] The configuration for determining the load or throughput of the information processing device 1B is arbitrary. For example, a monitor unit (not shown) may determine the load and notify each component. For example, when the information processing device 1B is a computer, the information processing device 1B may use a monitor of an operating system running on the computer as a configuration for determining the load. Alternatively, a predetermined application running on the computer may measure the processing speed of one or more components as the throughput. For example, the predetermined application may measure at least one of the quantity of inference target data processed by the primary inference unit 20B, the quantity of aggregated data generated by the data aggregation unit 30B, and the quantity of aggregated data processed by the secondary inference unit 40B in a unit time.

[0135] Further, in the information processing device 1B, the configuration for controlling the operation of each component in response to the load or throughput is arbitrary. For example, the information processing device 1B may include a control unit (not shown) that determines the load or throughput and controls each component. Alternatively, a predetermined configuration may determine the load or throughput and control the operation of components such as the primary inference unit 20B, the data aggregation unit 30B, and the secondary inference unit 40B. Alternatively, for example, the primary inference unit 20B, the data aggregation unit 30B, and the secondary inference unit 40B may determine the load or throughput and change the inference parameters related to each. Therefore, in the following description, except when particularly necessary, the description of the components that control the operation corresponding to the load in each component will be omitted.

[0136] Furthermore, the configurations to be subjected to load or throughput judgment and the configurations to switch inference parameters are arbitrary. For example, the load or throughput may be the load or throughput of the entire information processing device 1B. For example, the information processing device 1B may switch the inference parameters of the primary inference unit 20B, the data aggregation unit 30B, and the secondary inference unit 40B so that the throughput of the entire information processing device 1B becomes appropriate for the inference target data. Alternatively, the load or throughput may be judged to be the load or throughput of one or a part of the configurations of the information processing device 1B. For example, the information processing device 1B may switch the inference parameters of a certain configuration in response to the load or throughput of a configuration preceding or succeeding that configuration.

[0137] For example, the primary inference unit 20B may switch the trained model for primary inference in response to the load of the data aggregating unit 30B. For example, when the load of the data aggregating unit 30B is high, the primary inference unit 20B uses a trained model with a high load, that is, a trained model with high inference accuracy. Conversely, when the load of the data aggregating unit 30B is low, the primary inference unit 20B uses a trained model with a low load. Alternatively, the secondary inference unit 40B may switch the trained model for secondary inference in response to the load of the data aggregating unit 30B. For example, when the load of the data aggregating unit 30B is high, the secondary inference unit 40B uses a trained model with a high load, that is, a trained model with high inference accuracy. Conversely, when the load of the data aggregating unit 30B is low, the secondary inference unit 40B uses a trained model with a low load. Next, each configuration will be described. Note that the object inference unit 10 and the data acquisition unit 90 are similar to the object inference unit 10 and the data acquisition unit 90 of the second embodiment, and therefore detailed description will be omitted.

[0138] Like the data generation unit 60, the data generation unit 60B generates a primary inference training data set and a secondary inference training data set using an original training data set. Then, the data generation unit 60B stores the generated primary inference training data set and secondary inference training data set in the data storage unit 50B. However, in the third embodiment, at least one of the primary inference unit 20B and the secondary inference unit 40B uses multiple trained models. Therefore, the data generation unit 60B generates multiple training data sets as at least one of the primary inference training data set and the secondary inference training data set.

[0139] For example, the data generating unit 60B may apply a general data augmentation method to the original learning data set stored in the data storage unit 50B to generate multiple learning data sets. The data augmentation method is arbitrary. When the data is an image, the data augmentation method is, for example, left-right flipping, up-down flipping, partial cropping, combining, enlargement or reduction, brightness adjustment, luminance adjustment, color adjustment, and combinations thereof.

[0140] The data storage unit 50B stores a primary inference learning data set and a secondary inference learning data set, similar to the data storage unit 50. The primary inference learning data set and the secondary inference learning data set are learning data sets used to generate trained models used by the primary inference unit 20B and the secondary inference unit 40B, respectively. However, in the third embodiment, at least one of the primary inference unit 20B and the secondary inference unit 40B uses multiple trained models. Therefore, the data storage unit 50B stores multiple learning data sets as at least one of the primary inference learning data set and the secondary inference learning data set.

[0141] The model generation unit 70B generates a trained model for primary inference and a trained model for secondary inference using a training data set, similarly to the model generation unit 70. However, in the third embodiment, at least one of the primary inference unit 20B and the secondary inference unit 40B uses a plurality of trained models. Therefore, the model generation unit 70B generates a plurality of trained models as at least one of the trained models for primary inference and the trained models for secondary inference. Then, the model generation unit 70B stores the generated trained models in the model storage unit 80B. Specifically, the model generation unit 70B generates a plurality of trained models as at least one of the trained models for primary inference and the trained models for secondary inference using a plurality of training data sets stored in the data storage unit 50B. However, the model generation unit 70B may generate a plurality of trained models using one training data set.

[0142] The multiple trained models generated by the model generation unit 70B are arbitrary. For example, the model generation unit 70B may generate any one of the following trained models: (1) A training dataset used for training, (2) the network structure of the model; (3) Hyperparameters of the model, (4) The accuracy of the weights included in the model. (5) batch size in the model, (6) The positional relationship of the sub-object to the target object; (7) The number of sub-target objects for the target object. The model generation unit 70B may generate an index used for selecting a trained model from among the multiple trained models generated. For example, the model generation unit 70B may generate values ​​such as the processing load, throughput performance, scale, or size of the trained models generated or a comparison result of at least a part of these values ​​as an index for selecting a trained model.

[0143] The model storage unit 80B stores trained models, similar to the model storage unit 80. However, in the third embodiment, at least one of the primary inference unit 20B and the secondary inference unit 40B uses multiple trained models. Therefore, the model storage unit 80B stores multiple trained models as trained models for at least one of the trained models for primary inference and the trained models for secondary inference.

[0144] The data aggregator 30B generates aggregated data in the same manner as the data aggregator 30. However, the data aggregator 30B may change the data aggregation, including at least one of post-processing and data aggregation, in response to a predetermined load or throughput in the information processing device 1B. The change in data aggregation in the data aggregator 30B will be described in more detail later.

[0145] The primary inference unit 20B executes primary inference in the same manner as the primary inference unit 20. The secondary inference unit 40B executes secondary inference in the same manner as the secondary inference unit 40. Furthermore, at least one of the primary inference unit 20B and the secondary inference unit 40B switches the trained model used for inference in response to a predetermined load or throughput in the information processing device 1B. At least one of the primary inference unit 20B and the secondary inference unit 40B may switch between three or more trained models instead of two. When both the primary inference unit 20B and the secondary inference unit 40B switch the trained models, the primary inference unit 20B and the secondary inference unit 40B may use the same threshold value or different threshold values ​​as the threshold value used to determine the load or throughput.

[0146] The load and throughput used for the judgment in the third embodiment are arbitrary. An operator may determine the load or throughput used for the judgment based on the target object and the learned model, etc. For example, the information processing device 1B may use at least one of the following items as the load or throughput. The information processing device 1B may use a combination of two or more items instead of one item. When using the following values, the information processing device 1B may use statistical processing such as averaging. (1) Hardware resources related to computation constituting the information processing device 1B, (1-1) Utilization or operation rate of at least one of the CPU and ALU, (1-2) The length of a data queue for processing in at least one of the components of the information processing device 1B, or the processing waiting time, (1-3) A measurement value of a sensor that detects temperature or power consumption included in the information processing device 1B, (2) The size of the data to be inferred, (3) The result of the primary inference in the primary inference unit 20B, such as the number or area of ​​BBs to be post-processed, (4) The result of post-processing in the data aggregation unit 30B, such as the number or area of ​​BBs after post-processing (5) The result of aggregation in the data aggregator 30B, such as the size of the aggregated data, (6) The ratio of the size of the data to be inferred to the size of the aggregated data, (7) The result of the secondary inference in the secondary inference unit 40B, such as the number or area of ​​BBs included in the secondary inference, (8) Processing time of at least one of the primary inference and the secondary inference. Note that the processing time may be an actually measured time, or may be a time calculated using a preset relationship between the size of the inference target data and the processing time.

[0147] The information processing device 1B may switch the inference parameters based on other information instead of the load and throughput, or together with the load or throughput. For example, the shooting range of a surveillance camera of a system for detecting an intruder may include ranges with different required inference accuracy. For example, in a range near the entrance, it is desirable to have high inference accuracy. On the other hand, in a range far from the entrance, it is desirable to lower the load required for inference even if the inference accuracy is low in order to reduce unnecessary power consumption. Therefore, for example, the information processing device 1B usually uses inference parameters with low load. Note that the inference parameters with low load are, for example, learned models with low load. Then, for example, when a human presence sensor detects a person's intrusion into a predetermined range or approach to the vicinity of the entrance, the information processing device 1B may switch to inference parameters with high inference accuracy. For example, the inference parameters with high inference accuracy are learned models with high inference accuracy. In this way, the information processing device 1B may switch the inference parameters in response to predetermined information such as an event detected by a sensor.

[0148] The changes of the inference parameters in the primary inference unit 20B and the secondary inference unit 40B include the changes of the trained model in the above description, as follows. However, the changes of the inference parameters are not limited to the following. (1) Changing the trained model used for inference: When a trained model with a low load is used, the throughput of the primary inference unit 20B and the secondary inference unit 40B improves. However, the inference accuracy decreases. When a trained model with a high load but high accuracy is used, the throughput of the primary inference unit 20B and the secondary inference unit 40B decreases. However, the inference accuracy increases. (2) Changes in a given quantity that together perform first-order inference: When the predetermined number of times that the primary inferences are executed collectively is reduced, the information processing device 1B can shorten the time required to obtain a final inference result for the inference target data. The time required to obtain a final inference result is, for example, latency.

[0149] Next, a description will be given of changes to inference parameters (particularly data aggregation) related to the data aggregator 30B. The data aggregator 30B may change the following inference parameters based on a predetermined load or throughput in the information processing device 1B. (1) Changes in parameters for data aggregation, such as BB filtering or thresholds for BB adjustment or BB correction amounts: When the filtering threshold is changed so that the number of filtering targets is reduced, i.e., the amount of data after filtering is reduced, the throughput of data aggregation unit 30B improves. However, the inference accuracy of secondary inference unit 40B decreases. When the filtering threshold is changed so that the number of filtering targets is increased, i.e., the amount of data after filtering is increased, the throughput of data aggregation unit 30B decreases. However, the inference accuracy of secondary inference unit 40B improves. (2) Changing the gap between duplicated data in the aggregate: When the gap is narrowed, the throughput of the data aggregating unit 30B is improved. However, the inference accuracy of the secondary inference unit 40B is reduced. When the gap is widened, the throughput of the data aggregating unit 30B is reduced. However, the inference accuracy of the secondary inference unit 40B is improved. (3) How to group areas: (3-1) Change whether to combine overlapping and adjacent areas: When the regions are not combined, the throughput of the data aggregating unit 30B is improved. However, the inference accuracy of the secondary inference unit 40B is reduced. When the regions are combined, the throughput of the data aggregating unit 30B is reduced. However, the inference accuracy of the secondary inference unit 40B is improved. (3-2) Changes to the distance that determines proximity: When the distance for determining proximity is shortened, the throughput of the data aggregating unit 30B improves. However, the inference accuracy of the secondary inference unit 40B decreases. When the distance for determining proximity is lengthened, the throughput of the data aggregating unit 30B decreases. However, the inference accuracy of the secondary inference unit 40B improves. (4) Change in aggregate data size: When the size of the aggregated data is reduced, the throughput of the data aggregation unit 30B is improved. However, the inference accuracy of the secondary inference unit 40B is reduced. When the size of the aggregated data is increased, the throughput of the data aggregation unit 30B is reduced. However, the inference accuracy of the secondary inference unit 40B is improved.

[0150] The information processing device 1B may change items related to a plurality of components as follows as inference parameters. (1) Change in use of sub-objects: The information processing device 1B may switch whether or not to use a sub target object depending on the load. When the sub target object is not used, the throughput of the information processing device 1B is improved. However, the final inference accuracy is reduced. When the sub target object is used, the throughput of the information processing device 1B is reduced. However, the final inference accuracy is improved. (2) Presence or absence of first-order inference: The information processing device 1B may switch whether to use the primary inference unit 20B depending on the load. When the amount of aggregated data is large and primary inference is not used, the throughput of the information processing device 1B improves. When the amount of aggregated data is large and primary inference is used, the throughput of the information processing device 1B decreases. In this way, the switching of the primary inference included in the inference parameters includes whether or not to execute the primary inference. Note that the operation of the other components when primary inference is not used is optional. For example, when primary inference is not used, the data aggregation unit 30B may generate aggregated data using the inference target data. Alternatively, when primary inference is not used, the secondary inference unit 40B may infer a target object in the inference target data as secondary inference.

[0151] Next, the operation of the information processing device 1B according to the third embodiment will be described with reference to the drawings. In the following description, the average value of the quantity of aggregated data generated per unit time by the data aggregation unit 30B is used as an example of the load. Hereinafter, the average value of the quantity of aggregated data generated per unit time is referred to as the "aggregated data generation speed". In addition, the trained model for primary inference used by the primary inference unit 20B is used as the inference parameter to be changed. The information processing device 1B operates so as to obtain higher inference accuracy in a low load situation. Specifically, when the aggregated data generation speed is equal to or lower than a threshold, that is, when the load is low, the primary inference unit 20B uses a trained model with high accuracy, that is, a trained model with high load, for the primary inference. Conversely, when the aggregated data generation speed exceeds a threshold, that is, when the load is high, the primary inference unit 20B uses a trained model with low load, that is, a trained model with low accuracy, for the primary inference.

[0152] FIG. 12 is a flow diagram showing an example of a model switching operation in the information processing device 1B according to the third embodiment. FIG. 12 is an example of an operation in which the primary inference unit 20B switches a trained model for primary inference used for primary inference in response to the aggregated data generation speed. The information processing device 1B calculates the aggregated data generation speed (step S201). The information processing device 1B determines whether the aggregated data generation speed exceeds a predetermined threshold (step S202). If the aggregated data generation speed exceeds the threshold (Yes in step S202), the primary inference unit 20B executes primary inference using a trained model with a low load as the primary inference (step S205). If the aggregated data generation speed is equal to or lower than the threshold (No in step S202), the primary inference unit 20B executes primary inference using a trained model with high accuracy as the primary inference (step S206). The information processing device 1B repeats the above operation at a predetermined interval. Using such an operation, the information processing device 1B switches the trained model for primary inference used by the primary inference unit 20B in response to the throughput or load of the data aggregating unit 30B.

[0153] In this way, the information processing device 1B switches at least one of the inference parameters based on a predetermined load or throughput in the information processing device 1B. For example, the information processing device 1B switches at least one of the operations or data handled by the primary inference unit 20B, the secondary inference unit 40B, and the data aggregating unit 30B based on a predetermined load or throughput in the information processing device 1B. For example, the information processing device 1B switches at least one of the trained model for primary inference used by the primary inference unit 20B and the trained model for secondary inference used by the secondary inference unit 40B based on the load or throughput of the data aggregating unit 30B. Alternatively, the information processing device 1B switches the process of data aggregation in the data aggregating unit 30B based on the load or throughput of the primary inference unit 20B or the secondary inference unit 40B. In this way, the information processing device 1B according to the third embodiment changes the inference parameters according to a predetermined load or throughput in the information processing device 1B.

[0154] By using such operations, the information processing device 1B suppresses a decrease in throughput and avoids the occurrence of data that cannot be processed when the load increases. On the other hand, when the load decreases, the information processing device 1B switches the inference parameters so that the load increases. By using such operations, the information processing device 1B realizes inference with appropriate accuracy while ensuring throughput.

[0155] Although the present invention has been described above with reference to the embodiment, the present invention is not limited to the above embodiment. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Explanation of symbols]

[0156] 1. Information processing device 1B Information processing equipment 1C Information processing equipment 1D Data Processing Device 10 Object reasoning part 10D Object reasoning part 20 Primary reasoning part 20B Primary reasoning part 20D Primary inference part 30 Data Aggregation Department 30B Data Aggregation Department 30D Data Aggregation Department 40 Secondary reasoning part 40B Secondary reasoning part 40D Secondary inference part 50 Data Storage Section 50B Data storage section 60 Data Generation Division 60B Data Generation Section 70 Model Generation Unit 70B Model Generation Section 80 Model Storage Section 80B Model Storage Section 90 Data Acquisition Section 200 Data Acquisition Device 300 display device 400 Information Processing Systems 600 Computer equipment 610 CPU 611 ALU 620 ROM 630 RAM 640 Storage device 650NIC 690 Storage medium

Claims

1. A primary inference means for inferring the first target object by applying inference target data, at least a portion of which includes a first target object, to a first trained model as a primary inference; a data aggregation means for classifying the first target object inferred in the primary inference into any one of groups based on attributes of the first target object, generating aggregated data for each group, the aggregated data being data with a smaller amount than the inference target data, and generating a correspondence relationship between a position of the first target object in the inference target data and a position of the first target object in the aggregated data; A plurality of secondary inference means for inferring the first target object by applying the aggregated data of the groups different from each other to a second learned model as a secondary inference; an object inference means for inferring the first object in the inference target data by using the first object in the result of the secondary inference and the correspondence relationship; An information processing device comprising:

2. The secondary inference means use different trained models or different inference parameters. The information processing device according to claim 1 .

3. The data aggregating means estimates an inference difficulty for each of the secondary inference means, and classifies the first target object into one of the groups based on the inference difficulty.

3. The information processing device according to claim 1 or 2.

4. The primary inference means infers the inference difficulty level in each of the secondary inference means. The information processing device according to claim 3 .

5. An information processing device according to any one of claims 1 to 4, a data acquisition device that outputs the inference target data to the information processing device; a display device that acquires an inference result from the information processing device and displays the acquired inference result; An information processing system comprising:

6. An information processing device comprising: As a first inference, inference target data, at least a part of which includes a first target object, is applied to a first trained model to infer the first target object; classifying the first target object inferred in the primary inference into any one of groups based on attributes of the first target object, generating aggregated data for each group, the aggregated data being data with a smaller quantity than the inference target data, and generating a correspondence relationship between a position of the first target object in the inference target data and a position of the first target object in the aggregated data; As a secondary inference, the aggregated data of the different groups is applied to a second trained model to infer the first target object; Inferring the first object in the inference target data using the first object in the result of the secondary inference and the correspondence relationship. Information processing methods.

7. An information processing device executes the information processing method according to claim 6, a data acquisition device outputs the inference target data to the information processing device; A display device acquires the result of the inference from the information processing device and displays the acquired result of the inference. Information processing methods.

8. As a first inference, a process of applying inference target data, at least a part of which includes a first target object, to a first trained model to infer the first target object; a process of classifying the first target object inferred in the primary inference into any one of groups based on attributes of the first target object, generating aggregated data for each group, the aggregated data being data with a smaller amount than the inference target data, and generating a correspondence relationship between a position of the first target object in the inference target data and a position of the first target object in the aggregated data; As a secondary inference, a process of inferring the first target object by applying the aggregated data of the different groups to a second trained model; a process of inferring the first target object in the inference target data by using the first target object in the result of the secondary inference and the correspondence relationship; A program that causes a computer to execute the following.

Images