Information processing device, information processing system, information processing method, and program

Abstract

To improve throughput of target object inference.SOLUTION: An information processing apparatus of the present invention includes: primary inference means for inferring a first target object, by applying to a first trained model the inference target data in which at least a portion of the data includes a first target object, as primary inference; data aggregating means for generating aggregated data, which is data with a smaller quantity than the inference target data, using the first target object inferred in the primary inference, and generating a correspondence relationship between a position of the first target object in the inference target data and a position of the first target object in the aggregated data; secondary inference means for inferring the first target object by applying aggregated data to a second trained model as secondary inference; and object inference means for inferring the first target object in the inference target data, using a first target object in a result of secondary inference and the correspondence relationship.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to processing information, and in particular to inference using machine learning. [Background technology]

[0002] One of the main tasks using machine learning is the object detection task in an image. The object detection task is a task to generate a list of pairs of positions and classes (types) of target objects present in an image. In recent years, object detection tasks using deep learning, especially machine learning, have been widely used (for example, see Non-Patent Documents 1 to 3).

[0003] In the learning phase of machine learning, the object detection task is given a group of learning images and information on the target object in each image as correct answer data. The information on the target object is selected according to the specifications of the object detection task. For example, the information on the target object includes the coordinates of the four vertices of a rectangular area in which the target object is reflected (bounding box (BB)) and the class of the target object. Note that the following description will also use the BB and class as an example of the information on the target object. Then, the object detection task uses the group of learning images and the information on the target object to generate a trained model as a result of machine learning using, for example, deep learning.

[0004] In the detection phase, the object detection task applies the trained model to an image including a target object to infer the target object contained in the image, and outputs a pair of BB and class for each target object contained in the image. Note that the object detection task may also output an evaluation result (e.g., confidence) for the object detection result together with the BB and class.

[0005] For example, a surveillance system for people and cars can be constructed by inputting images from a surveillance camera into an object detection task and using the positions and classes of people and cars in the surveillance camera images detected by the object detection task. The object detection task may also be used in combination with other recognition processes. For example, a license plate recognition system can be constructed by combining an object detection task that detects license plates of cars with an OCR (Optical Character Recognition) process that recognizes characters on the detected license plates. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", [online], January 6, 2016, Cornel University, [Retrieved March 1, 2021], Internet<URL:https: / / arxiv.org / abs / 1506.01497> [Non-Patent Document 2] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C. Berg, "SSD: Single Shot MultiBox Detector", [online], December 29, 2016, Cornel University, [Retrieved March 1, 2021], Internet,<URL:https: / / arxiv.org / abs / 1512.02325> [Non-Patent Document 3] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollar, "Focal Loss for Dense Object Detection", [online], International Conference on Computer Vision (ICCV), 2017, pp. 2980-2988, [Retrieved March 1, 2021], Internet,<URL: https: / / openaccess.thecvf.com / content_ICCV_2017 / papers / Lin_Focal_Loss_for_ICCV_2017_paper.pdf> Summary of the Invention [Problem to be solved by the invention]

[0007] The processing of object detection tasks requires high throughput. For example, in order to prevent oversight, surveillance cameras need to capture images at a relatively high frame rate. In particular, when targeting fast-moving objects such as cars, surveillance cameras are required to operate at a frame rate of, for example, 10 fps (frames per second) or more. Therefore, the object detection task requires the detection of objects in images captured by the surveillance camera at a high frame rate.

[0008] Furthermore, in a system for detecting an object in an image captured by a surveillance camera, a configuration is assumed in which the surveillance camera or a device in the periphery of the surveillance camera (i.e., the edge) executes inference processing for the object detection task. However, in the edge, there are often restrictions on installation space, cooling, power, etc., and only limited computing resources are available. Even in such an edge environment, a certain level of high throughput is required for the processing of the object detection task.

[0009] However, inference processing using machine learning, particularly processing of an object detection task, is a process that requires a high computation load and a long processing time. As a result, the object detection task generally has low throughput performance. Therefore, it is desired to improve the throughput of inference processing in the object detection task.

[0010] Non-Patent Documents 1 to 3 disclose techniques related to object detection tasks using deep learning. However, Non-Patent Documents 1 to 3 do not disclose techniques related to improving the throughput of inference processing. In other words, Non-Patent Documents 1 to 3 have a problem in that they cannot improve the throughput of object detection tasks.

[0011] An object of the present invention is to provide an information processing device or the like that solves the above problems and improves the throughput of inference of a target object. [Means for solving the problem]

[0012] According to one aspect of the present invention, there is provided an information processing device comprising: a primary inference means for inferring the first target object by applying inference target data, at least a portion of which includes a first target object, to a first trained model as a primary inference; a data aggregation means for generating aggregated data, which is data having a smaller quantity than the inference target data, using a first object inferred in the primary inference, and generating a correspondence relationship between the position of the first object in the inference target data and the position of the first object in the aggregated data; A secondary inference means for inferring a first target object by applying the aggregated data to a second trained model as a secondary inference; an object inference means for inferring a first object in the inference target data by using a first object in the result of the secondary inference and the correspondence relationship; Includes.

[0013] An information processing system according to one embodiment of the present invention includes: The above information processing device; A data acquisition device that outputs inference target data to an information processing device; a display device that acquires the inference result from the information processing device and displays the acquired inference result; Includes.

[0014] An information processing method according to one aspect of the present invention includes: As a first inference, inference target data, at least a portion of which includes a first target object, is applied to a first trained model to infer the first target object; Using the first object inferred in the primary inference, generate aggregated data, which is data having a smaller quantity than the inference target data, and generate a correspondence relationship between the position of the first object in the inference target data and the position of the first object in the aggregated data; As a second inference, applying the aggregated data to a second trained model to infer the first object of interest; Using the first object in the result of the secondary inference and the correspondence relationship, a first object in the inference target data is inferred.

[0015] An information processing method according to one aspect of the present invention includes: An information processing device executes the above information processing method, The data acquisition device outputs inference target data to the information processing device; The display device acquires the result of inference from the information processing device and displays the acquired result of inference.

[0016] A program according to one embodiment of the present invention includes: A process of applying inference target data, at least a portion of which includes a first target object, to a first trained model as a first inference to infer the first target object; A process of generating aggregated data, which is data having a smaller quantity than the inference target data, using the first object inferred in the primary inference, and a process of generating a correspondence relationship between the position of the first object in the inference target data and the position of the first object in the aggregated data; A process of inferring a first target object by applying the aggregated data to a second trained model as a secondary inference; A process of inferring a first object in the inference target data using the first object in the result of the secondary inference and the correspondence relationship; to be executed by the computer. Effect of the Invention

[0017] According to the present invention, it is possible to obtain an effect of improving the throughput of inference of a target object. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a block diagram illustrating an example of a configuration of an information processing device according to the first embodiment. [Diagram 2] FIG. 2 is a flow diagram illustrating an example of an operation of generating a trained model in the information processing device according to the first embodiment. [Diagram 3] FIG. 3 is a flow diagram illustrating an example of an operation of inferring a target object in the information processing device according to the first embodiment. [Figure 4] FIG. 4 is a flow diagram illustrating an example of post-processing in the data aggregation unit. [Diagram 5] FIG. 5 is a diagram showing an example of a result of the primary inference. [Figure 6] FIG. 6 is a diagram illustrating an example of the aggregated data. [Figure 7] FIG. 7 is a diagram showing an example of a result of the secondary inference. [Figure 8] FIG. 8 is a diagram showing an example of inference of a target object in inference target data. [Figure 9] FIG. 9 is a block diagram illustrating an example of a hardware configuration of an information processing device. [Figure 10] FIG. 10 is a block diagram showing an example of a configuration of an information processing system including an information processing device. [Figure 11] FIG. 11 is a block diagram illustrating an example of a configuration of an information processing device according to the second embodiment. [Figure 12]FIG. 12 is a flow diagram illustrating an example of a model switching operation in the information processing device according to the second embodiment. [Figure 13] FIG. 13 is a block diagram illustrating an example of a configuration of an information processing device according to the third embodiment. As illustrated in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Each drawing is for explaining an embodiment of the present invention. However, each embodiment is not limited to the description in each drawing. In addition, the same reference numerals are used to designate similar configurations in each drawing, and repeated description thereof may be omitted.

[0020] In the following description, the following will be used as an example for each embodiment. (1) Data to be inferred: Images (still images or videos) taken by a camera, etc. (2) Inference task: Object detection task to infer the target object in an image; (3) Inference result: the set of “BB, class, and confidence” (4) Target object: vehicle license plate (License Plate: LP).

[0021] However, these do not limit each embodiment. For example, each embodiment may use data other than an image as data to be processed. For example, each embodiment may use depth data acquired using a depth sensor as data to be processed. Alternatively, each embodiment may use three-dimensional data acquired using Light Detection and Ranging (LIDAR) as data to be processed. Alternatively, the result of inference in each embodiment is not limited to the set of "BB, class, and confidence". For example, each embodiment may infer a target object included in the BB. In the following description, for convenience of explanation, the set of "BB, class, and confidence" may be simply called "BB".

[0022] <First embodiment> [Configuration Description] The configuration of the first embodiment will be described with reference to the drawings. Fig. 1 is a block diagram showing an example of the configuration of an information processing device 1 according to the first embodiment. The information processing device 1 includes an object inference unit 10, a primary inference unit 20, a data aggregation unit 30, a secondary inference unit 40, a data storage unit 50, a data generation unit 60, a model generation unit 70, a model storage unit 80, and a data acquisition unit 90.

[0023] The number of components and the connection relationships shown in Fig. 1 are an example. For example, the information processing device 1 may include a plurality of data acquisition units 90. The information processing device 1 may be configured using a computer device including a CPU (Central Processing Unit), a main memory, and a secondary storage device. In this case, the components of the information processing device 1 shown in Fig. 1 indicate functions realized using the CPU, etc. The hardware configuration will be described further later.

[0024] In the information processing device 1, the configuration for controlling the operation of each component is arbitrary. For example, the information processing device 1 may include a control unit (not shown) that controls each component. Alternatively, a predetermined configuration (for example, the object inference unit 10) may control the operation of components such as the primary inference unit 20, the data aggregation unit 30, and the secondary inference unit 40. Alternatively, each component may independently acquire data from other components and operate. Alternatively, each component may start operating when it acquires data from other components. Therefore, in the following description, a description of the control of the operation of the components will be omitted unless particularly necessary.

[0025] (1) Data Acquisition Unit 90 The data acquisition unit 90 acquires data (hereinafter referred to as "inference target data") including an object to be inferred (hereinafter referred to as "target object") from a specific device. At least a portion of the inference target data needs to include the target object. In other words, at least a portion of the inference target data does not need to include the target object. The device from which the data acquisition unit 90 acquires the inference target data is arbitrary. For example, the data acquisition unit 90 may acquire images from a surveillance camera as the inference target data.

[0026] (2) Primary reasoning section 20 The primary inference unit 20 infers a target object (e.g., an LP) in the inference target data. Hereinafter, the inference in the primary inference unit 20 is referred to as a "primary inference". In addition to the target object (e.g., an LP), the primary inference unit 20 may infer an object (e.g., a peripheral object of the LP) that is in a predetermined positional relationship with the target object as a target of the primary inference. Hereinafter, an object that is in a predetermined positional relationship with the target object is referred to as a "sub-target object". In the following description, the primary inference unit 20 infers a target object and a sub-target object. However, this does not limit the operation of the primary inference unit 20. The primary inference unit 20 does not have to infer a sub-target object. In addition, in the following description, the target object of inference in the information processing device 1 may be referred to as a "first target object", and the sub-target object may be referred to as a "second target object".

[0027] The predetermined positional relationship is arbitrary. An operator may set the positional relationship of the sub-target objects in accordance with the target object. For example, one example of the positional relationship is that "the target object is contained in the sub-target object" or "the sub-target object contains the target object." For example, when the target object is the LP of a vehicle, the sub-target objects are the front and rear surfaces of the vehicle that contain the LP (hereinafter, collectively referred to as "vehicle front and rear surfaces"). However, this does not limit the positional relationship in the first embodiment.

[0028] The primary inference unit 20 uses machine learning in the primary inference. In detail, the primary inference unit 20 infers a target object and sub-target objects in the inference target data using a model (hereinafter referred to as a "trained model") generated using machine learning and stored in the model storage unit 80. Hereinafter, the trained model used by the primary inference unit 20 is referred to as a "trained model for primary inference" or a "first trained model."

[0029] The primary inference unit 20 may modify the inference target data when applying the inference target data to the primary inference trained model. For example, if the inference target data is an image, the primary inference unit 20 may change the size or aspect ratio of the image.

[0030] (3) Data Aggregation Unit 30 The data aggregating unit 30 executes the process described below using the target object included in the result of the primary inference. However, when the primary inference unit 20 infers a sub-target object in addition to the target object, the data aggregating unit 30 may use the sub-target object in addition to the target object. In the following description, the data aggregating unit 30 uses the target object and the sub-target object. However, the data aggregating unit 30 does not have to use the sub-target object. In this case, the data aggregating unit 30 may omit the operations related to the sub-target object in the following description.

[0031] The data aggregating unit 30 performs a predetermined process (e.g., discarding the BB, filtering the BB, or adjusting the BB) on the target objects included in the result of the primary inference. Hereinafter, this process will be referred to as "post-processing". The data aggregating unit 30 may perform post-processing on some of the target objects included in the result of the primary inference, or may perform post-processing on all of the target objects. Furthermore, the data aggregating unit 30 may perform post-processing on some or all of the sub-target objects included in the result of the primary inference. In the following description, as an example, the data aggregating unit 30 performs post-processing on all target objects and sub-target objects. However, the post-processing in the data aggregating unit 30 is not limited to this.

[0032] Furthermore, the data aggregation unit 30 generates data (hereinafter referred to as "aggregated data") that includes the regions of the target object and the regions of the sub-target objects in the inference target data (e.g., images of the target object and the sub-target objects) and has a smaller amount of data than the inference target data. Hereinafter, this process will be referred to as "aggregation." In other words, "aggregation" means extracting (or duplicating) the regions that include the target object and the sub-target objects in the result of the primary inference from the inference target data, collecting the extracted (or duplicated) regions, and generating aggregated data that is a smaller amount of data than the inference target data.

[0033] The quantity of data that is less in the aggregated data compared to the inference target data is arbitrary. The operator may decide it according to the target object and the inference operation. For example, when the number of data is used as the quantity of data, the aggregated data is data with a smaller quantity of data compared to the inference target data. For example, when the data is still images, the number of data is the number of still images. Or, when the data is videos of a fixed duration, the number of data is, for example, the number of videos. When the amount of data is used as the quantity of data, the aggregated data is data with a smaller amount of data (e.g., capacity or area) compared to the inference target data. For example, when the data is still images, the amount of data is the amount of data of the entire still images. Or, when the data is videos, the amount of data is the duration of the videos.

[0034] Furthermore, in generating the aggregated data, the data aggregating unit 30 generates a correspondence relationship (hereinafter referred to as an "aggregated correspondence relationship") between the positions of the target object and the sub-target object in the inference target data and the positions of the target object and the sub-target object in the aggregated data. For example, when inferring BB, the data aggregating unit 30 generates, as the aggregated correspondence relationship, a coordinate transformation between the position, orientation, and size of the target object BB and the sub-target object BB in the inference target data and their respective positions, orientations, and sizes in the aggregated data.

[0035] In addition, when the data is an image, the data aggregating unit 30 may generate an image in the aggregated data having a size different from that of the image in the inference target data. For example, an operator of the information processing device 1 may set the size of the image in the aggregated data to the data aggregating unit 30 based on the size of the inference target data, the performance of the hardware of the information processing device 1, and the processing performance and calculation accuracy of the trained model used for inference. The aggregated data is data used for inference by the secondary inference unit 40 described later. Moreover, the inference target data is data used for inference by the primary inference unit 20. In other words, the information processing device 1 may use data of different sizes (e.g., images of different sizes) between the primary inference unit 20 and the secondary inference unit 40.

[0036] Next, the details of post-processing and aggregation in the data aggregating unit 30 will be described. Note that "post-processing" and "aggregation" in the data aggregating unit 30 are collectively referred to as "data aggregation." In other words, the "data aggregation" operation includes the "post-processing" operation before the generation of aggregated data and the "aggregation" operation. However, the data aggregating unit 30 does not have to execute "post-processing." In this case, the data aggregation becomes the "aggregation" operation in the following description.

[0037] (3-1) Post-processing The data aggregating unit 30 executes post-processing described below before generating aggregated data. For example, the data aggregating unit 30 discards unnecessary sub-target object regions (e.g., BB) from the result of the primary inference using the positional relationship between the target object and the sub-target object and the result of the primary inference. For example, the data aggregating unit 30 discards sub-target objects whose target objects in a predetermined positional relationship with the sub-target object are included in the result of the primary inference. Alternatively, the data aggregating unit 30 may discard BBs of sub-target objects (e.g., front and rear surfaces of a vehicle) corresponding to a target object (e.g., LP) inferred as a result of the primary inference. In other words, the data aggregating unit 30 uses sub-target objects whose target objects in a predetermined positional relationship are not included in the result of the primary inference as targets for aggregation from the sub-target objects included in the result of the primary inference. The reason for using this operation will be explained using "positional relationship in which a sub-target object includes a target object" and "BB".

[0038] The BB of the target object included in the result of the primary inference is included in the aggregated data in the aggregation operation described later. Therefore, even if the BB of the sub-target object including the target object included in the result of the primary inference is not included in the aggregated data, the BB of the target object is included in the aggregated data. Furthermore, if the BB of the sub-target object is included in the aggregated data, the aggregated data will include a duplicated BB as the BB of the target object. This will result in an unnecessary load being added in the secondary inference. Therefore, the data aggregation unit 30 discards such a sub-target object. As a result, the information processing device 1 can improve the throughput when using the sub-target object. On the other hand, the BB of the sub-target object not including the target object in the result of the primary inference may include a target object that was not inferred in the primary inference. Therefore, if the BB of such a sub-target object is included in the aggregated data, the information processing device 1 can improve the accuracy of the secondary inference.

[0039] Therefore, the data aggregating unit 30 discards, among the sub-target objects, any sub-target object whose target object is included in the result of the primary inference. In other words, the data aggregating unit 30 uses, as a target for aggregation, among the sub-target objects having a predetermined positional relationship with the target object, any sub-target object whose target object having a predetermined positional relationship is not included in the result of the primary inference. As a result of this operation, the data aggregating unit 30 can reduce the amount of aggregated data generated while improving the inference accuracy in the secondary inference. As a result, the information processing device 1 can improve the throughput (and the inference speed).

[0040] For example, when the positional relationship is "the vehicle front and rear surfaces (sub-target object) encompass the LP (target object)," the data aggregating unit 30 discards the BBs of the vehicle front and rear surfaces included in the result of the primary inference that include the LP included in the result of the primary inference within the area of ​​the BB. In other words, the data aggregating unit 30 aggregates the BBs of the vehicle front and rear surfaces that do not have an inferred LP within their area among the BBs of the vehicle front and rear surfaces included in the result of the primary inference. In this way, the data aggregating unit 30 generates aggregated data so as not to include the result of the primary inference in which both the target object (e.g., the LP) and the sub-target object (e.g., the vehicle front and rear surfaces) are inferred.

[0041] The positional relationship between the target object and the sub-target object is not limited to the above and is arbitrary. For example, the data aggregating unit 30 determines the positional relationship between the target object and the sub-target object using the positional relationship set by the operator. In addition, in determining the positional relationship between the target object and the sub-target object, the data aggregating unit 30 may use a trained model that determines the positional relationship between the target object and the sub-target object, which is generated using a predetermined machine learning in a device not shown.

[0042] The data aggregating unit 30 may execute an operation different from discarding the sub-target object. For example, when a target object is detected and a sub-target object corresponding to the target object is detected, the data aggregating unit 30 may update the confidence of the BB of the target object without discarding the BB of the sub-target object. For example, in this case, the data aggregating unit 30 may add or multiply the confidence of the BB of the target object by a predetermined value, or may change the value of the confidence to a predetermined value.

[0043] The data aggregating unit 30 may filter the result of the primary inference as a post-processing before generating aggregated data. The target of filtering is arbitrary. For example, the data aggregating unit 30 may filter the BB of the result of the primary inference using at least one of the confidence corresponding to the BB, the size or aspect ratio of the BB, and the position of the BB in the image. More specifically, for example, the data aggregating unit 30 may filter the BB based on a comparison between a predetermined threshold and an attribute value (e.g., size or aspect ratio) of the BB. For example, the data aggregating unit 30 may discard a BB with a size or aspect ratio that is inappropriate for inference. Note that the threshold may not be one, but may be multiple. For example, the threshold may be different for each class related to the BB. Alternatively, the threshold may be different depending on whether a sub-target object corresponding to the target object is detected or not.

[0044] Alternatively, the data aggregation unit 30 may adjust at least one of the size and aspect ratio of the acquired BB (BB included in the result of the primary inference). The position and size of the BB included in the result of the primary inference may differ from the correct position and size. For example, the BB size may be smaller than the actual target object. In particular, when the trained model used in the primary inference is small and lightweight, the tendency for the inferred BB to be smaller than the actual target object becomes noticeable.

[0045] If the BB, whose size is smaller than the actual target object, is used as is, the data aggregation unit 30 generates aggregated data including the target object with a missing part. As a result, the secondary inference unit 40, which will be described later, will perform inference using aggregated data including the target object with a missing part. As a result, the possibility that the result of inference in the secondary inference unit 40 will be incorrect (the possibility that the inference accuracy will be low) increases. Therefore, as a post-processing, the data aggregation unit 30 may correct the size of the BB by adding or multiplying a predetermined value to at least one of the height and width of the BB of the target object and the BB of the sub-target object. Alternatively, if the aspect ratio of the BB deviates from a predetermined range, the data aggregation unit 30 may correct at least one of the height and width of the BB so that the aspect ratio falls within the predetermined range. For example, the data aggregation unit 30 may use different values ​​as the amount of correction of the BB, corresponding to at least one of the BB size, class, position in the image, and relative position with respect to a predetermined object in the image. Alternatively, in the case of a BB of a target object, the data aggregating unit 30 may use different values ​​as the amount of correction of the BB depending on whether a sub-target object corresponding to the target object is detected or not, or depending on the size of the BB of the corresponding sub-target object, etc. In these cases, the operator may set in advance the respective values ​​used by the data aggregating unit 30.

[0046] It should be noted that the data aggregating unit 30 does not have to execute "post-processing". However, whether or not post-processing is executed, the data aggregating unit 30 executes the same operation as the aggregation operation described below. Therefore, for the sake of convenience in the following description, the operation described below will be described as the operation "after post-processing", including the case where post-processing is not executed. In other words, in the following description, the operation "after post-processing" includes the operation "when post-processing is not executed".

[0047] (3-2) Aggregation The data aggregating unit 30 generates aggregated data that collects the regions of the target object and sub-target objects in the result of the primary inference after post-processing (for example, the region including the BB of the target object and the region including the BB of the sub-target object). In detail, the data aggregating unit 30 extracts (or duplicates) the region of the target object and the region of the sub-target object in the result of the primary inference after post-processing of the inference target data, and generates aggregated data that collects the extracted (or duplicated) regions.

[0048] The data aggregating unit 30 may generate one aggregated data from multiple inference target data, or may generate multiple aggregated data from multiple inference target data. However, even in the case of multiple aggregated data, the amount of aggregated data is less than the amount of inference target data. If the inference target data is an image, the data aggregating unit 30 may provide a predetermined gap (spacing) between the images when generating the aggregated data. As already explained, the data aggregating unit 30 may perform aggregation without performing post-processing. In this case, the data aggregating unit 30 performs aggregation using the result of the primary inference.

[0049] Furthermore, the data aggregating unit 30 may combine at least one of duplicated primary inference results and adjacent primary inference results into one in the primary inference results after post-processing of the inference target data, and then aggregate them. For example, the data aggregating unit 30 may aggregate duplicated BBs and BBs within a predetermined distance into a BB that includes all of these BBs. By using such an operation, the information processing device 1 can reduce the possibility of double detection of the same object in the secondary inference and the possibility of overlooking the target object.

[0050] In addition, the data aggregating unit 30 may change (e.g., enlarge or reduce) the size of the target object and sub-target objects in the result of the primary inference (e.g., BB) after post-processing of extraction (or duplication) before aggregating. When the size of the target object and sub-target object is reduced, the amount of aggregated data generated is reduced. As a result, the secondary inference time is more likely to be shortened (i.e., the throughput is more likely to be improved). On the other hand, when the size of the target object and sub-target object is increased, the size of the target object, etc. in the generated aggregated data (i.e., the size of the target object, etc. in the secondary inference) is increased. As a result, the inference accuracy of the secondary inference is more likely to be improved.

[0051] The method of changing the size of the target object and the sub-target object is arbitrary. For example, the data aggregating unit 30 may determine whether to change the size and, if so, how to change it based on the size of the target object and the sub-target object in the result of the primary inference after post-processing (e.g., comparison with a threshold value). Alternatively, the data aggregating unit 30 may determine whether to change the size and, if so, how to change it using the result of the primary inference (e.g., at least one of the position and size, class, and confidence of the BB).

[0052] The data aggregating unit 30 may perform a predetermined image processing (e.g., a complementing process) in changing the size. Alternatively, the data aggregating unit 30 may perform a predetermined image processing (e.g., a brightness adjustment, a luminance adjustment, a color adjustment, a contrast adjustment, or a geometric correction) in addition to or instead of changing the size. Furthermore, the data aggregating unit 30 may adjust a parameter related to the image processing in response to the result of the primary inference (e.g., at least one of the position and size, class, and confidence of the BB).

[0053] Furthermore, the data aggregating unit 30 generates a correspondence relationship (aggregation correspondence relationship) between the area of ​​the target object in the inference target data (e.g., the position, orientation, and size of the BB) and the area of ​​the target object in the aggregated data (e.g., the position, orientation, and size of the BB). When aggregating the areas of the sub-target objects, the data aggregating unit 30 similarly generates a correspondence relationship (aggregation correspondence relationship) between the area of ​​the sub-target object in the inference target data and the area of ​​the sub-target object in the aggregated data for the area of ​​the sub-target object.

[0054] (4) Secondary inference section 40 The secondary inference unit 40 infers a target object in the aggregated data generated by the data aggregating unit 30. Hereinafter, the inference in the secondary inference unit 40 is referred to as "secondary inference". The secondary inference unit 40 uses machine learning in the secondary inference. In detail, the secondary inference unit 40 infers a target object in the aggregated data using a trained model stored in the model storage unit 80. Hereinafter, the trained model used by the secondary inference unit 40 is referred to as a "trained model for secondary inference" or a "second trained model". For example, the secondary inference unit 40 uses the trained model for secondary inference to infer a set of "BB, class, and confidence" of a target object included in the aggregated data as a result of secondary inference.

[0055] The secondary inference unit 40 may change the size of the aggregated data when applying the aggregated data to the trained model for secondary inference. For example, if the aggregated data is an image, the secondary inference unit 40 may change the size or aspect ratio of the image in the aggregated data before applying it to the trained model for secondary inference.

[0056] (5) Data generation unit 60 The data generating unit 60 generates a learning dataset to be used in machine learning using a predetermined dataset (hereinafter referred to as an "original learning dataset" or a "first dataset"). Any data may be included in the original learning dataset as long as it is data corresponding to a machine learning technique. For example, the original learning dataset is a group of images including correct answer data related to a target object. The source of the original learning dataset is any. For example, an operator may store the original learning dataset in the data storage unit 50 in advance. Alternatively, the data generating unit 60 may acquire the original learning dataset from a device not shown in the figure when generating the learning dataset.

[0057] In detail, the data generating unit 60 generates a training data set (hereinafter referred to as a "primary inference training data set" or a "first training data set") used in machine learning to generate a trained model for primary inference used by the primary inference unit 20. Furthermore, the data generating unit 60 generates a training data set (hereinafter referred to as a "secondary inference training data set" or a "second training data set") used in machine learning to generate a trained model for secondary inference used by the secondary inference unit 40.

[0058] For example, the data generating unit 60 generates correct answer data of the target object and the sub-target object based on the correct answer data of the target object included in the original learning data set and the information on the positional relationship between the target object and the sub-target object. Then, the data generating unit 60 generates a primary inference learning data set by adding the generated correct answer data to the original learning data set. For example, the data generating unit 60 generates a BB (correct answer data of the front and rear surfaces of the vehicle) that is larger than the BB of the LP included in the original learning data set by multiplying the height and width by a predetermined value. Then, the data generating unit 60 adds the generated correct answer data of the front and rear surfaces of the vehicle to the original learning data set to generate a primary inference learning data set. Alternatively, the data generating unit 60 applies a predetermined trained model to the original learning data set as the correct answer data of the LP to infer the BB of the LP. Then, the data generating unit 60 generates a BB that is larger than the BB of the inferred LP by multiplying the height and width by a predetermined value as the correct answer data of the front and rear surfaces of the vehicle. Then, the data generating unit 60 generates a primary inference training dataset by adding the generated LP and the correct answer data of the front and rear surfaces of the vehicle to the original training dataset. Alternatively, the data generating unit 60 may apply a predetermined trained model to the original training dataset as the correct answer data to infer the BB of the LP and the BB of the front and rear surfaces of the vehicle, and add the inferred BB to the original training dataset to generate a primary inference training dataset.

[0059] Alternatively, for example, the data generating unit 60 generates aggregated data from the supervised data of the original training dataset using the data aggregating unit 30. Then, the data generating unit 60 generates a secondary inference training dataset by adding the supervised data and the aggregated data to the original training dataset. Note that the data generating unit 60 may generate the secondary inference training dataset by using a primary inference training dataset instead of the original training dataset, or a dataset obtained by adding the primary inference training dataset to the original training dataset.

[0060] When the primary inference unit 20 and the secondary inference unit 40 use the same trained model, the data generating unit 60 may generate one training data set as the primary inference training data set and the secondary inference training data set. Alternatively, the data generating unit 60 may generate the primary inference training data set and the secondary inference training data set using different original training data sets.

[0061] The data generation unit 60 then stores the generated primary inference learning data set and secondary inference learning data set in the data storage unit 50. However, the data generation unit 60 may acquire at least one of the primary inference learning data set and secondary inference learning data set from an external device (not shown). In this case, the data generation unit 60 stores the acquired primary inference learning data set or secondary inference learning data set in the data storage unit 50.

[0062] (6) Data storage unit 50 The data storage unit 50 stores a primary inference learning dataset and a secondary inference learning dataset. Furthermore, the data storage unit 50 may store an original learning dataset. The source of the original learning dataset is arbitrary. For example, an operator may store the original learning dataset in the data storage unit 50 in advance.

[0063] The information processing device 1 may use an external storage device (not shown) as the data storage unit 50. In this case, the information processing device 1 may not include the data storage unit 50 as a physical configuration. Alternatively, the information processing device 1 may acquire at least one of a primary inference learning dataset and a secondary inference learning dataset generated in a device (not shown) or the like, and store it in the data storage unit 50. When acquiring both the primary inference learning dataset and the secondary inference learning dataset, the information processing device 1 may not include the data generation unit 60.

[0064] (7) Model generation unit 70 The model generation unit 70 performs machine learning on a predetermined model using a data set for primary inference learning stored in the data storage unit 50, to generate a trained model for primary inference (first trained model) used by the primary inference unit 20. Furthermore, the model generation unit 70 performs machine learning on a predetermined model using a data set for secondary inference learning stored in the data storage unit 50, to generate a trained model for secondary inference (second trained model) used by the secondary inference unit 40. Note that the source of acquisition of the model used to generate the trained model is arbitrary. For example, an operator may store the model in the model storage unit 80 in advance. Alternatively, the model generation unit 70 may acquire the model used to generate the trained model from a device not shown in the figure in generating the trained model.

[0065] The model generation unit 70 may generate models having the same model type, network structure, and size as the trained model for primary inference and the trained model for secondary inference, or may generate different models. For example, the model generation unit 70 may generate the same model as the trained model for primary inference and the trained model for secondary inference. That is, the model generation unit 70 may generate one model as the trained model for primary inference and the trained model for secondary inference. For example, when the primary inference unit 20 and the secondary inference unit 40 use the same trained model, the model generation unit 70 may generate one trained model as the trained model for primary inference and the trained model for secondary inference. Alternatively, the model generation unit 70 may generate models that are different from each other as the trained model for primary inference and the trained model for secondary inference but have the same model type, network structure, and size as the trained model for primary inference and the trained model for secondary inference.

[0066] Alternatively, the model generation unit 70 may generate models that differ in at least one of the model type, network structure, and size, etc., as the trained model for primary inference and the trained model for secondary inference. For example, when generating both trained models using deep learning, the model generation unit 70 may generate models that differ in at least one of the model type, network structure, and size, as the trained model for primary inference and the trained model for secondary inference. For example, the model generation unit 70 may generate a trained model that is small and lightweight as the trained model for primary inference, compared to the trained model for secondary inference. However, this does not limit the trained models generated by the model generation unit 70.

[0067] The machine learning technique used by the model generation unit 70 is arbitrary. For example, the model generation unit 70 may use deep learning to generate at least one of the trained model for primary inference and the trained model for secondary inference. The model generation unit 70 may use the same machine learning for generating the trained model for primary inference and the trained model for secondary inference, or may use different machine learning. Note that in the description of the first embodiment, as an example, the model generation unit 70 generates different trained models as the trained model for primary inference and the trained model for secondary inference.

[0068] Then, the model generation unit 70 stores the generated trained model for primary inference and trained model for secondary inference in the model storage unit 80. However, the model generation unit 70 may acquire at least one of the trained model for primary inference and the trained model for secondary inference from an external device not shown. In this case, the model generation unit 70 stores the acquired trained model for primary inference or trained model for secondary inference in the model storage unit 80.

[0069] (8) Model storage unit 80 The model storage unit 80 stores the trained model for primary inference and the trained model for secondary inference. The model storage unit 80 may store a model used by the model generation unit 70 to generate the trained model. For example, an operator may store a model used to generate the trained model in the model storage unit 80 in advance.

[0070] The information processing device 1 may use an external storage device (not shown) as the model storage unit 80. In this case, the information processing device 1 may not include the model storage unit 80 as a physical configuration. Alternatively, the information processing device 1 may acquire at least one of a trained model for primary inference and a trained model for secondary inference generated in a device (not shown) or the like, and store the acquired model in the model storage unit 80. When acquiring both a trained model for primary inference and a trained model for secondary inference, the information processing device 1 may not include the data storage unit 50, the data generation unit 60, and the model generation unit 70.

[0071] (9) Object reasoning part 10 The object inference unit 10 infers the target object in the inference target data by using the result of the secondary inference in the secondary inference unit 40 (the target object in the aggregated data) and the aggregated correspondence generated by the data aggregation unit 30 in generating the aggregated data. For example, when the coordinate transformation of BB is stored as the aggregated correspondence, the object inference unit 10 applies the inverse transformation of the coordinate transformation to the coordinates of BB inferred as a result of the secondary inference as the inference position of BB, to calculate the position of BB of the target object in the inference target data. For example, when the rectangle of BB obtained as the result of the secondary inference protrudes from the rectangle in the aggregated data (the rectangle generated based on BB obtained as a result of the primary inference), the object inference unit 10 may discard the BB. Alternatively, for example, when the value (quotient) obtained by dividing the area of ​​the intersection part of the rectangle of BB obtained as a result of the secondary inference and the rectangle on the aggregated data by the area of ​​the rectangle of BB obtained as a result of the secondary inference is less than a predetermined threshold, the object inference unit 10 may discard the BB. If the quotient is equal to or greater than a predetermined threshold, the object inference unit 10 may modify the rectangle BB to the intersection portion.

[0072] [Explanation of operation] (A) Generation of trained models 2 is a flow diagram showing an example of an operation of generating a trained model in the information processing device 1 according to the first embodiment. The information processing device 1 starts generating a trained model when a predetermined condition is met. For example, the information processing device 1 starts generating a trained model when an instruction from an operator is received. In this case, at the start of the operation, the information processing device 1 may acquire parameters required for generating the trained model from the operator.

[0073] The parameters required for generating a trained model are arbitrary. For example, when the data storage unit 50 stores a plurality of training data sets, the parameters are information indicating which training data set is to be used. Alternatively, the parameters may be parameters related to the machine learning process in the model generation unit 70. Note that the information processing device 1 may acquire other information in addition to the parameters. For example, the data generation unit 60 may acquire at least a part of the training data set from an operator. In this case, the data generation unit 60 stores the acquired training data set in the data storage unit 50.

[0074] The data generation unit 60 generates a primary inference learning dataset using the original learning dataset stored in the data storage unit 50 (step S100). For example, the data generation unit 60 generates a primary inference learning dataset using supervised answer data included in the original learning dataset. Then, the data generation unit 60 stores the primary inference learning dataset in the data storage unit 50. The model generation unit 70 performs machine learning in a predetermined model using the primary inference learning dataset to generate a trained model for primary inference (step S101). Then, the model generation unit 70 stores the trained model for primary inference in the model storage unit 80.

[0075] The data generation unit 60 generates a secondary inference learning dataset using the original learning dataset stored in the data storage unit 50 (step S102). For example, the data generation unit 60 may generate aggregated data using the supervised data of the original learning dataset, and generate a secondary inference learning dataset including the supervised data and the aggregated data. Then, the data generation unit 60 stores the generated secondary inference learning dataset in the data storage unit 50. The model generation unit 70 executes machine learning in a predetermined model using the secondary inference learning dataset, and generates a trained model for secondary inference (step S103). Then, the model generation unit 70 stores the trained model for secondary inference in the model storage unit 80.

[0076] After generating the trained model, the information processing device 1 may notify the operator of the execution result.

[0077] (B) Inference behavior 3 is a flow diagram showing an example of an operation of inferring a target object in the information processing device 1 according to the first embodiment. Note that before the inference operation starts, the model storage unit 80 stores a trained model. However, if the model storage unit 80 does not store a trained model, the information processing device 1 may start the inference operation after performing the operation of "(A) Generation of trained model" to generate a trained model.

[0078] The information processing device 1 starts inference of a target object when a predetermined condition is met. For example, the information processing device 1 starts inference of a target object when an instruction from an operator is met. The information processing device 1 may acquire parameters (e.g., designation of a trained model to be used) from an operator in inferring a target object. However, the information processing device 1 may use parameters given in advance. Alternatively, the information processing device 1 may automatically start inference of a target object after starting the device. Alternatively, the information processing device 1 may execute the operation described below when the data acquisition unit 90 acquires one or a predetermined amount of inference target data. However, the information processing device 1 may start the inference operation asynchronously with the acquisition of the inference target data in the data acquisition unit 90. Note that the data acquisition unit 90 may acquire the inference target data in advance and store it in a storage unit not shown. In this case, the primary inference unit 20 may acquire the inference target data from the storage unit.

[0079] The primary inference unit 20 executes primary inference using the inference target data (step S114). The primary inference unit 20 may collectively execute primary inference on a predetermined amount of inference target data. Hereinafter, the predetermined amount of inference target data on which the primary inference unit 20 collectively executes primary inference is referred to as an "inference target data group." For example, when the inference target data is an image including a target object, the primary inference unit 20 infers, as primary inference, a set of "BB, class, and confidence" pairs from the images of the predetermined amount of inference target data (inference target data group). Furthermore, the primary inference unit 20 may infer sub-target objects in the inference target data group.

[0080] The primary inference unit 20 may determine the inference target data group to be inferred collectively using a condition other than the quantity of the inference target data. For example, the primary inference unit 20 may determine the inference target data group to be inferred collectively using at least one of the following conditions: (a) When the number of BBs inferred in the first inference reaches a predetermined number, (b) When the sum of the sizes of BBs inferred in the first inference reaches a predetermined value, (c) inferring the number of aggregated data, and when the inferred number of aggregated data reaches a predetermined value, (d) Inference target data for which a predetermined time has passed since it was acquired by the information processing device 1. However, the primary inference unit 20 may execute primary inference for all of the acquired inference target data. When executing primary inference for an inference target data group, the primary inference unit 20 may process the primary inference for at least a part of the inference target data included in the inference target data group in parallel, may process all of the primary inferences sequentially, or may combine parallel processing and sequential processing.

[0081] FIG. 5 is a diagram showing an example of a result of the primary inference. In FIG. 5, of the two rectangles, the smaller inner rectangle is the BB of the target object (LP) obtained as a result of the primary inference. The larger outer rectangle is the BB of the sub-target object (front and rear surfaces of the vehicle) obtained as a result of the primary inference. Note that FIG. 5 omits the display of classes and confidences. Returning to the explanation with reference to FIG. 3.

[0082] The data aggregating unit 30 performs post-processing on some or all of the results of the primary inference (step S115). For example, the data aggregating unit 30 performs filtering and adjustment of the BB using the attributes of the BB and information on the positional relationship between the target object and the sub-target object. FIG. 4 is a flow diagram showing an example of the post-processing in the data aggregating unit 30. More specifically, FIG. 4 shows an example of the operation of the post-processing in the data aggregating unit 30 when the BB is inferred as a result of the primary inference. Hereinafter, with reference to FIG. 4, the operation when the target object is the LP and the sub-target object is the front and rear surfaces of the vehicle including the LP will be described.

[0083] The data aggregation unit 30 judges whether the BB is a BB of the target object (LP) (step S131). Note that the case where the BB is not a BB of the target object (LP) is the case where the BB is a BB of a sub-target object (front and rear surfaces of the vehicle).

[0084] If the BB is not the BB of the target object (LP) (No in step S131), the data aggregating unit 30 judges whether the sub-target object (front and rear surfaces of the vehicle) has a predetermined relationship with the target object (LP) included in the result of the primary inference (step S132). For example, the data aggregating unit 30 judges whether the BB of the sub-target object (front and rear surfaces of the vehicle) includes the BB of the target object (LP) included in the result of the primary inference.

[0085] In the case of the BB of the target object (LP) (Yes in step S131), or if the sub-target object (front and rear surfaces of the vehicle) does not have a predetermined relationship (No in step S132), the data aggregator 30 judges whether the BB has a shape appropriate for use as aggregated data (step S133). For example, the data aggregator 30 judges whether the size and aspect ratio of the BB are within a predetermined threshold range (e.g., whether they are not too small).

[0086] If the BB has an appropriate shape (Yes in step S133), the data aggregator 30 modifies the BB to a shape appropriate for generating aggregated data (step S134). For example, the data aggregator 30 sets the size and aspect ratio of the BB to values ​​appropriate for generating aggregated data. For example, the data aggregator 30 may change the size of the BB by multiplying the height and width of the BB by a predetermined value. Alternatively, the data aggregator 30 may modify the height and width of the BB so that the aspect ratio of the BB falls within a predetermined range.

[0087] If the sub-target objects have a predetermined relationship (Yes in step S132) or if the BB does not have an appropriate shape (No in step S133), the data aggregating unit 30 discards the BB (step S135). That is, the data aggregating unit 30 does not include the BB in the aggregated data. In this way, the data aggregating unit 30 performs post-processing on the result of the primary inference. Returning to the explanation with reference to FIG. 3.

[0088] The data aggregating unit 30 generates aggregated data using all the results of the primary inference after post-processing (step S116). For example, when the inference target data is an image, the data aggregating unit 30 generates an image that collects images of the target object region (BB) and the sub-target object region (BB) inferred as the result of the primary inference. Note that the data aggregating unit 30 may generate aggregated data using a region that combines at least a part of overlapping regions and adjacent regions into one in the result of the primary inference.

[0089] FIG. 6 is a diagram showing an example of aggregated data. In FIG. 6, the display of classes and confidences is omitted. The aggregated data in FIG. 6 is an example of aggregated data generated using the data in FIG. 5 and four pieces of data (a total of five pieces of data) that are temporally before and after FIG. 5. Furthermore, the aggregated data in FIG. 6 is an example of a case where an LP is detected in all of the five pieces of data in the first inference. In addition, if an LP is not detected in any of the pieces of data and the front and rear surfaces of the vehicle are detected, the image corresponding to that data is an image of the front and rear surfaces of the vehicle rather than the LP. In the example of FIG. 6, the data aggregation unit 30 makes the area of ​​the image to be extracted larger than the area of ​​the BB of the result of the first inference in order to reduce the possibility that a part of the target object is missing. For example, the image of each LP is an image of an area larger than the BB of the LP. In the example shown in FIG. 6, the total area of ​​the images of the result of the first inference is smaller than the area of ​​the image of the aggregated data. Therefore, the image of the aggregated data in FIG. 6 includes a surplus area (the black area on the right side of FIG. 6). For example, when further aggregating the results of the primary inference, the data aggregating unit 30 may add at least a part of the results of the primary inference to be aggregated to the surplus area. If the results of the primary inference to be added do not fit in the surplus area, the data aggregating unit 30 may generate new aggregated data.

[0090] Note that the sub-target object in Fig. 5 includes the target object included in the result of the primary inference. In other words, the sub-target object in Fig. 5 is a sub-target object that has a predetermined positional relationship with the target object. Therefore, the sub-target object BB in Fig. 5 is not included in the aggregated data in Fig. 6. Returning to the explanation with reference to Fig. 3.

[0091] Furthermore, when generating the aggregated data, the data aggregation unit 30 generates a correspondence relationship (aggregation correspondence relationship) between the positions of the data areas of the target object and sub-target objects included in the inference target data and the positions of the data areas of the target object and sub-target objects in the aggregated data.

[0092] The secondary inference unit 40 executes secondary inference using the aggregated data (step S118). Fig. 7 is a diagram showing an example of the result of secondary inference. The rectangle in Fig. 7 is BB of the target object (LP) inferred by the secondary inference unit 40. Note that Fig. 7 omits the display of classes and confidences. Returning to the explanation with reference to Fig. 3.

[0093] The object inference unit 10 infers a target object in the inference target data using the result of the secondary inference and the aggregated correspondence relationship (step S119). FIG. 8 is a diagram showing an example of inference of a target object in the inference target data. Compared with FIG. 5, the position of BB in FIG. 8 is an appropriate position with respect to LP. Note that, if the size of a target object included in the result of the secondary inference is larger than the size of the data aggregated in the aggregated data, the object inference unit 10 may ignore the target object. Returning to the explanation with reference to FIG. 3.

[0094] The information processing device 1 repeats the above operations until a predetermined condition is satisfied. For example, the information processing device 1 may end the operations when it receives an end instruction from an operator. Alternatively, the information processing device 1 may repeat the above operations until there is no inference target data to be processed, until the amount of processed inference target data reaches a predetermined value, until a predetermined time has elapsed, or until the number of repetitions reaches a predetermined number. Based on the above operations, the information processing device 1 infers a target object in the inference target data.

[0095] [Effect description] The information processing device 1 according to the first embodiment can achieve the effect of improving the throughput of inference of a target object for the following reasons.

[0096] The information processing device 1 includes an object inference unit 10, a primary inference unit 20, a data aggregation unit 30, and a secondary inference unit 40. As a primary inference, the primary inference unit 20 applies inference target data, at least a part of which includes a first target object, to a first learned model (a learned model for primary inference) to infer the first target object. The data aggregation unit 30 uses the first target object inferred in the primary inference to generate aggregated data, which is data with a smaller amount than the inference target data. Furthermore, the data aggregation unit 30 generates a correspondence relationship (aggregated correspondence relationship) between the position of the first target object in the inference target data and the position of the first target object in the aggregated data. As a secondary inference, the secondary inference unit 40 applies the aggregated data to a second learned model (a learned model for secondary inference) to infer the first target object. The object inference unit 10 infers the first target object in the inference target data using the first target object in the result of the secondary inference and the correspondence relationship (aggregated correspondence relationship).

[0097] In this way, the information processing device 1 generates aggregated data that aggregates areas including the target object in the inference target data, using the result of the primary inference using the inference target data. Then, the information processing device 1 infers the target object in the inference target data, using the result of the secondary inference using the aggregated data and information related to the positions of the inference target data and the aggregated data of the target object (aggregation correspondence).

[0098] The aggregated data is data that collects areas of target objects in the inference target data, and is smaller in quantity than the inference target data. Therefore, the load of the secondary inference is lower than the load when all of the inference target data is used. Furthermore, the primary inference only needs to be able to infer the target object with a certain degree of accuracy. In other words, the information processing device 1 can use an inference that is lower in load than the secondary inference as the primary inference. As a result, the information processing device 1 can improve the throughput in inference.

[0099] Using a specific example, the improvement of the inference throughput in the information processing device 1 will be described. As an inference to be compared with the information processing device 1, an inference in which a model equivalent to a trained model for secondary inference is applied to inference target data (hereinafter referred to as "associated inference") is used. Note that in the following description, the following is assumed as a premise. (1) The data to be inferred is an image. (2) The performance (e.g., processing speed and accuracy) and scale of the association inference are the same as the performance and scale of the secondary inference in the secondary inference unit 40. The time required for the association inference and secondary inference to infer one image is unit time (hereinafter, the unit time is defined as "1"). (3) The time required for the primary inference in the primary inference unit 20 to infer one image is one-tenth (0.1) of that required for the association inference and secondary inference. (4) The data aggregating unit 30 generates, on average, one aggregated image from three images to be inferred (the ratio of the number of images to be inferred to the number of images of the aggregation result is 3:1). (5) The total number of images is N. (6) Processing other than inference is ignored because its processing time is small compared to inference.

[0100] In this case, the inference time of the association inference is "N×1=N". The inference time of the primary inference in the information processing device 1 is "N×0.1=0.1N". The inference time of the secondary inference is "N / 3×1=N / 3". That is, the inference time of the information processing device 1 is "approximately 0.43N (=0.1N+N / 3)". In this way, the information processing device 1 can make inference in a shorter time than the association inference. As a result, the information processing device 1 can improve the throughput compared to the association inference. In addition, the scale of the model (trained model for secondary inference) used for the final inference (secondary inference) in the information processing device 1 is equivalent to the model used for the association inference. Therefore, the inference accuracy of the information processing device 1 is equivalent to that of the association inference.

[0101] In addition, a lightweight model tends to have a lower accuracy at high speed compared to a heavy model. For example, in the above example, the accuracy of the primary inference in the primary inference unit 20 tends to be lower than the accuracy of the secondary inference in the secondary inference unit 40. Therefore, the primary inference unit 20 may infer a second target object (sub-target object) that is in a predetermined positional relationship with a first target object (target object of inference in the information processing device 1). In this case, the data aggregation unit 30 may generate aggregated data and a correspondence relationship using the first target object and the second target object (sub-target object). Using the above operation, the information processing device 1 can execute inference using not only the target object (first target object) of inference in the information processing device 1, but also a sub-target object (second target object) that is in a predetermined relationship with the target object. Therefore, the information processing device 1 can improve the accuracy of inference. For example, the information processing device 1 can suppress overlooking of the target object.

[0102] Furthermore, the data aggregating unit 30 may use a second target object (sub-target object) that is not included in the result of the primary inference and that is in a predetermined positional relationship with the first target object, among the second target objects (sub-target objects) included in the result of the primary inference. Using the above operations, the information processing device 1 can reduce the size of the aggregated data. As a result, the information processing device 1 can improve the throughput.

[0103] Furthermore, the data aggregating unit 30 may perform a predetermined process (post-processing) on ​​at least a part of the first target object included in at least the result of the primary inference before generating the aggregated data. Using the above operation, the information processing device 1 can generate more appropriate aggregated data. As a result, the information processing device 1 further improves the accuracy of inference. For example, the information processing device 1 suppresses overlooking of the target object. Note that, when a sub-target object is used, the information processing device 1 may perform a predetermined process (post-processing) on ​​at least a part of the second target object (sub-target object) included in the result of the primary inference before generating the aggregated data.

[0104] Further, the information processing device 1 includes a data generation unit 60 and a model generation unit 70. The data generation unit 60 uses a first dataset (original learning dataset) to generate a first learning dataset used for learning a first learned model (learned model for primary inference). Furthermore, the data generation unit 60 uses at least one of the first dataset and the first learning dataset to generate a second learning dataset used for learning a second learned model (learned model for secondary inference). The model generation unit 70 uses the first learning dataset to generate a first learned model (learned model for primary inference) and uses the second learning dataset to generate a second learned model (learned model for secondary inference). Using the above configuration, the information processing device 1 can generate a learned model used for inference.

[0105] [Variations] In the above description, an example has been given in which there is one primary inference unit 20 and one secondary inference unit 40. However, the configuration of the information processing device 1 is not limited to this. For example, the information processing device 1 may include multiple primary inference units 20. In this case, each of the multiple primary inference units 20 may perform inference using different parameters (e.g., a trained model, a data size, or a recognition target class). Furthermore, all of the multiple primary inference units 20 may perform inference using the same inference target data. Alternatively, at least some of the primary inference units 20 may perform inference using inference target data different from that of the other primary inference units 20.

[0106] Alternatively, for example, the information processing device 1 may include two or more secondary inference units 40. When there are multiple secondary inference units 40, at least some of the secondary inference units 40 may perform inference using parameters (e.g., a trained model or a data size) different from those of the other secondary inference units 40. When there are multiple secondary inference units 40, the data aggregating unit 30 may generate aggregated data corresponding to each secondary inference unit 40. When generating aggregated data corresponding to each of the multiple secondary inference units 40, the data aggregating unit 30 may generate aggregated data using different parameters for each secondary inference unit 40.

[0107] Alternatively, the data aggregating unit 30 may divide the generated aggregated data and allocate it to multiple secondary inference units 40. When allocating to multiple secondary inference units 40, the data aggregating unit 30 may generate or divide the aggregated data using, for example, the following attribute of BB in the result of the primary inference or a combination of these attributes. Note that the data aggregating unit 30 allocates the aggregated data to multiple secondary inference units 40 when the data aggregating unit 30 generates aggregated data corresponding to each of the multiple secondary inference units 40, or when the data aggregating unit 30 divides the aggregated data generated by the data aggregating unit 30. (1) BB size or aspect ratio, (2) BB's class, (3) BB confidence, (4) BB's position in the results of the first-order inference, (5) If BB is the BB of the target object, whether the corresponding sub-target object is detected or not.

[0108] The data aggregation unit 30 may include a part of the result of the primary inference (for example, BB) in the multiple aggregated data to be generated or the multiple aggregated data after division. The data aggregation unit 30 may use the above-mentioned attributes or a combination thereof to determine whether or not to include the result of the primary inference in the multiple aggregated data. When the result of the primary inference is included in the multiple aggregated data, the multiple secondary inference units 40 may infer the same target object. In this case, the object inference unit 10 may determine the overlap of the target object using the result of the secondary inference and the aggregated correspondence relationship, and execute an operation corresponding to the overlap using the determination result. For example, the object inference unit 10 may correct the result of the inference using the overlap. For example, the object inference unit 10 may select, as the BB of the target object, a BB with the highest confidence among the multiple BBs for the target object that are inferred in an overlapping manner. Alternatively, the object inference unit 10 may average the rectangular information of the BB in the multiple inference results as the rectangular information of the BB of the target object. For the averaging, the object inference unit 10 may use a weighted average based on the confidence.

[0109] In the above description, the data aggregating unit 30 corrects the size of the BB by adding or multiplying at least one of the height and width of the BB of the target object and the BB of the sub-target object by a predetermined value. However, the first embodiment is not limited to this. As another example, the data aggregating unit 30 may correct the size of the BB by multiplying at least one of the height and width of the BB of the target object and the BB of the sub-target object by a predetermined value and then adding a predetermined value. Alternatively, the data aggregating unit 30 may correct the size of the BB by applying a predetermined linear or nonlinear transformation to at least one of the height and width of the BB of the target object and the BB of the sub-target object. In this case, the parameters related to the transformation may be given to the data aggregating unit 30 in advance, for example. Alternatively, the data aggregating unit 30 may switch the transformation method and the parameters related to the transformation based on the attributes of the BB and the inference target data, or other information (for example, the load of the information processing device 1).

[0110] [Hardware configuration] In the above description, an example has been described in which the object inference unit 10, the primary inference unit 20, the data aggregation unit 30, the secondary inference unit 40, the data storage unit 50, the data generation unit 60, the model generation unit 70, the model storage unit 80, and the data acquisition unit 90 are included in one device. However, the first embodiment is not limited to this. For example, the information processing device 1 may be configured by connecting devices having functions corresponding to each configuration via a predetermined network. For example, the information processing device 1 may be realized using cloud computing. Alternatively, each component of the information processing device 1 may be configured by a hardware circuit. Alternatively, in the information processing device 1, multiple components may be configured by one hardware.

[0111] Alternatively, the information processing device 1 may be realized as a computer device including a CPU, a read only memory (ROM), and a random access memory (RAM). The information processing device 1 may be realized as a computer device including a network interface circuit (NIC) in addition to the above configuration. Alternatively, the information processing device 1 may be realized as a computer device including an arithmetic unit (ALU) that executes some or all of the operations of learning and inference. FIG. 9 is a block diagram showing an example of a hardware configuration of the information processing device 1. The information processing device 1 in FIG. 9 includes a CPU 610, an ALU 611, a ROM 620, a RAM 630, a storage device 640, and a NIC 650, and constitutes a computer device.

[0112] The CPU 610 reads a program from the ROM 620 and / or the storage device 640. Then, the CPU 610 controls the RAM 630, the storage device 640, the ALU 611, and the NIC 650 based on the read program. The CPU 610 controls these components to realize the functions of the object inference unit 10, the primary inference unit 20, the data aggregating unit 30, the secondary inference unit 40, the data storage unit 50, the data generation unit 60, the model generation unit 70, the model storage unit 80, and the data acquisition unit 90.

[0113] When implementing each function, the CPU 610 may use the RAM 630 or the storage device 640 as a temporary storage medium for the program. The CPU 610 may also read, using a storage medium reading device (not shown), a program contained in a storage medium 690 that stores a computer-readable program. Alternatively, the CPU 610 may receive a program from an external device (not shown) via the NIC 650, store the program in the RAM 630 or the storage device 640, and operate based on the stored program.

[0114] The ALU 611 is responsible for predetermined arithmetic processing and part of predetermined processing in the CPU 610. For example, the ALU 611 is controlled by the CPU 610 to execute some or all of the calculations of learning and inference. The configuration of the ALU 611 is arbitrary. For example, the ALU 611 may be a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). Alternatively, the ALU 611 may be, for example, an ASIC (Application Specific Integrated Circuit). Note that information (data, programs, circuit information, etc.) required for the calculations in the ALU 611 is stored in advance in the ROM 620, the RAM 630, or the storage device 640.

[0115] The ROM 620 stores programs and fixed data executed by the CPU 610. The ROM 620 is, for example, a P-ROM (Programmable-ROM) or a flash ROM. The RAM 630 temporarily stores programs and data executed by the CPU 610. The RAM 630 is, for example, a D-RAM (Dynamic-RAM). The storage device 640 stores data and programs that the information processing device 1 stores for a long period of time. The storage device 640 may operate as a data storage unit 50. The storage device 640 may also operate as a model storage unit 80. The storage device 640 may also operate as a temporary storage device for the CPU 610. The storage device 640 may operate as a secondary storage device for the memory (for example, the ROM 620 and the RAM 630). The storage device 640 is, for example, a hard disk device, a magneto-optical disk device, a solid state drive (SSD), or a disk array device.

[0116] The ROM 620 and the storage device 640 are non-volatile (non-transitory) recording media. On the other hand, the RAM 630 is a volatile (transitory) recording medium. The CPU 610 can operate based on a program stored in the ROM 620, the storage device 640, or the RAM 630. In other words, the CPU 610 can operate using a non-volatile recording medium or a volatile recording medium.

[0117] The NIC 650 relays data exchange with external devices (not shown) via a network. The NIC 650 operates as a part of the data acquisition unit 90 and the object inference unit 10. The NIC 650 is, for example, a LAN (Local Area Network) card. The NIC 650 is not limited to being wired, and may be wireless.

[0118] The information processing device 1 in Fig. 9 configured as above can obtain the same effect as the information processing device 1 in Fig. 1. This is because the CPU 610 of the information processing device 1 in Fig. 9 can realize the same function as the information processing device 1 in Fig. 1 based on a program. Alternatively, the CPU 610 and ALU 611 of the information processing device 1 in Fig. 9 can realize the same function as the information processing device 1 in Fig. 1 based on a program.

[0119] [system] 10 is a block diagram showing an example of a configuration of an information processing system 400 including an information processing device 1. The information processing system 400 includes an information processing device 1, a data acquisition device 200, and a display device 300. The information processing system 400 may include a plurality of devices as each device. For example, the information processing system 400 may include a plurality of data acquisition devices 200. Note that the information processing system 400 may include a device not shown. For example, the information processing system 400 may include a device (e.g., an OCR device) that recognizes predetermined information related to a target object using an inference result of the information processing device 1.

[0120] The data acquisition device 200 outputs inference target data to the information processing device 1. The data acquisition device 200 may output an original learning data set for generating a learning data set to the information processing device 1. The data acquisition device 200 is, for example, a surveillance camera. In this case, the data acquisition device 200 outputs a captured image to the information processing device 1 as the inference target data.

[0121] The information processing device 1 operates as described above. That is, the information processing device 1 acquires inference target data from a predetermined device (e.g., the data acquisition device 200). Then, the information processing device 1 infers a target object in the acquired inference target data (e.g., a set of "BB, class, and confidence" of the target object). Then, the information processing device 1 outputs the inference result to a predetermined device (e.g., the display device 300).

[0122] The display device 300 acquires an inference result (for example, a set of "BB, class, and confidence" of the target object) from the information processing device 1, and displays the acquired inference result. The display device 300 is, for example, a liquid crystal display, an organic electroluminescence display, or electronic paper. Specifically, the display device 300 is, for example, a liquid crystal display of a surveillance system. An operator can check the target object by referring to the display of the display device 300.

[0123] Thus, the information processing system 400 includes the information processing device 1, the data acquisition device 200, and the display device 300. The information processing device 1 operates as described above. The data acquisition device 200 outputs inference target data to the information processing device 1. The display device 300 acquires an inference result from the information processing device 1, and displays the acquired inference result. Using the above configuration, the information processing system 400 displays the target object inferred in the inference target data to an operator or the like.

[0124] <Second embodiment> The load of the primary inference and the secondary inference differs depending on the trained model used for inference. For example, a trained model with high inference accuracy generally has a high load. However, when a trained model with a high load is used, the throughput decreases. Alternatively, the load of data aggregation differs depending on the post-processing and aggregation processing in data aggregation. Alternatively, the method of generating aggregated data used in data aggregation (e.g., how to group regions) and the format of the aggregated data (e.g., size and gaps between images) affect the inference accuracy.

[0125] In this way, a change in at least one of the primary inference, secondary inference, and data aggregation (hereinafter collectively referred to as "inference parameters") affects the throughput and inference accuracy in the information processing device 1. Generally, there is a trade-off between inference accuracy and throughput. Furthermore, increasing the processing load decreases the throughput, and decreasing the processing load increases the throughput. Therefore, in order to prevent a decrease in inference accuracy while maintaining the desired throughput, it is desirable to select and use appropriate inference parameters based on the load or throughput.

[0126] Therefore, as a second embodiment, an embodiment in which at least one of the inference parameters (primary inference, secondary inference, and data aggregation) is changed based on the load or throughput will be described. Hereinafter, the second embodiment will be described with reference to the drawings. In the description of the second embodiment, the same configurations and operations as those of the first embodiment are given the same reference numerals, and detailed description thereof may be omitted. In the following description, the second embodiment changes the trained model as a change in the inference parameters. However, this does not limit the second embodiment. In the second embodiment, an inference parameter different from the trained model may be changed according to the load or throughput. The change in the inference parameters will be described further later.

[0127] [Configuration Description] Fig. 11 is a block diagram showing an example of the configuration of an information processing device 1B according to the second embodiment. The information processing device 1B includes an object inference unit 10, a primary inference unit 20B, a data aggregation unit 30B, a secondary inference unit 40B, a data storage unit 50B, a data generation unit 60B, a model generation unit 70B, a model storage unit 80B, and a data acquisition unit 90. The information processing device 1B may be configured using a computer device as shown in Fig. 9, similar to the first embodiment.

[0128] The configuration for determining the load or throughput of the information processing device 1B is arbitrary. For example, a monitor unit (not shown) may determine the load and notify each component. For example, when the information processing device 1B is a computer, the information processing device 1B may use a monitor of an operating system running on the computer as a configuration for determining the load. Alternatively, a predetermined application running on the computer may measure the processing speed of one or more components as the throughput. For example, the predetermined application may measure at least one of the quantity of inference target data processed by the primary inference unit 20B, the quantity of aggregated data generated by the data aggregation unit 30B, and the quantity of aggregated data processed by the secondary inference unit 40B in a unit time.

[0129] Further, in the information processing device 1B, the configuration for controlling the operation of each component in response to the load or throughput is arbitrary. For example, the information processing device 1B may include a control unit (not shown) that determines the load or throughput and controls each component. Alternatively, a predetermined configuration may determine the load or throughput and control the operation of components such as the primary inference unit 20B, the data aggregation unit 30B, and the secondary inference unit 40B. Alternatively, each component (for example, the primary inference unit 20B, the data aggregation unit 30B, and the secondary inference unit 40B) may determine the load or throughput and change the inference parameters related to each. Therefore, in the following description, except when particularly necessary, a description of the components that control the operation in response to the load in each component will be omitted.

[0130] Furthermore, the configurations to be subjected to load or throughput judgment and the configurations to switch inference parameters are arbitrary. For example, the load or throughput may be the load or throughput of the entire information processing device 1B. For example, the information processing device 1B may switch the inference parameters of the primary inference unit 20B, the data aggregation unit 30B, and the secondary inference unit 40B so that the throughput of the entire information processing device 1B becomes appropriate for the inference target data. Alternatively, the load or throughput may be judged to be the load or throughput of one or a part of the configurations of the information processing device 1B. For example, the information processing device 1B may switch the inference parameters of a certain configuration in response to the load or throughput of a configuration preceding or succeeding that configuration.

[0131] For example, the primary inference unit 20B may switch the trained model for primary inference in response to the load of the data aggregating unit 30B. For example, when the load of the data aggregating unit 30B is high, the primary inference unit 20B uses a trained model with a high load (a trained model with high inference accuracy). Conversely, when the load of the data aggregating unit 30B is low, the primary inference unit 20B uses a trained model with a low load. Alternatively, the secondary inference unit 40B may switch the trained model for secondary inference in response to the load of the data aggregating unit 30B. For example, when the load of the data aggregating unit 30B is high, the secondary inference unit 40B uses a trained model with a high load (a trained model with high inference accuracy). Conversely, when the load of the data aggregating unit 30B is low, the secondary inference unit 40B uses a trained model with a low load.

[0132] Next, each component will be described. Note that the object inference unit 10 and the data acquisition unit 90 are the same as those in the first embodiment, and therefore detailed description thereof will be omitted.

[0133] Like the data generating unit 60, the data generating unit 60B generates a primary inference training data set and a secondary inference training data set using an original training data set. Then, the data generating unit 60B stores the generated primary inference training data set and secondary inference training data set in the data storage unit 50B. However, in the second embodiment, at least one of the primary inference unit 20B and the secondary inference unit 40B uses multiple trained models. Therefore, the data generating unit 60B generates multiple training data sets as at least one of the primary inference training data set and the secondary inference training data set.

[0134] For example, the data generating unit 60B may apply a general data augmentation method to the original learning data set stored in the data storage unit 50B to generate multiple learning data sets. The data augmentation method is arbitrary. When the data is an image, the data augmentation method is, for example, left-right flipping, up-down flipping, partial cropping, combining, enlargement or reduction, brightness adjustment, luminance adjustment, color adjustment, and combinations thereof.

[0135] Like the data storage unit 50, the data storage unit 50B stores training data sets (primary inference training data set and secondary inference training data set) used to generate trained models used by the primary inference unit 20B and the secondary inference unit 40B. However, in the second embodiment, at least one of the primary inference unit 20B and the secondary inference unit 40B uses multiple trained models. Therefore, the data storage unit 50B stores multiple training data sets as at least one of the primary inference training data set and the secondary inference training data set.

[0136] The model generation unit 70B generates trained models (trained models for primary inference and trained models for secondary inference) using a training data set, similar to the model generation unit 70. However, in the second embodiment, at least one of the primary inference unit 20B and the secondary inference unit 40B uses multiple trained models. Therefore, the model generation unit 70B generates multiple trained models as at least one of the trained models for primary inference and the trained models for secondary inference. Then, the model generation unit 70B stores the generated trained models in the model storage unit 80B. Specifically, the model generation unit 70B generates multiple trained models as at least one of the trained models for primary inference and the trained models for secondary inference using multiple training data sets stored in the data storage unit 50B. However, the model generation unit 70B may generate multiple trained models using one training data set.

[0137] The multiple trained models generated by the model generation unit 70B are arbitrary. For example, the model generation unit 70B may generate any one of the following trained models: (1) A training dataset used for training, (2) the network structure of the model; (3) Hyperparameters of the model, (4) The accuracy of the weights included in the model. (5) batch size in the model, (6) The positional relationship of the sub-object to the target object; (7) The number of sub-target objects for the target object.

[0138] The model generation unit 70B may generate an index used for selecting a trained model from among the multiple trained models generated. For example, the model generation unit 70B may generate values ​​such as the processing load, throughput performance, scale, or size of the trained models generated or a comparison result of at least a part of these values ​​as an index for selecting a trained model.

[0139] The model storage unit 80B stores trained models, similar to the model storage unit 80. However, in the second embodiment, at least one of the primary inference unit 20B and the secondary inference unit 40B uses multiple trained models. Therefore, the model storage unit 80B stores multiple trained models as trained models for at least one of the trained models for primary inference and the trained models for secondary inference.

[0140] The data aggregator 30B generates aggregated data in the same manner as the data aggregator 30. However, the data aggregator 30B may change data aggregation (at least one of post-processing and data aggregation processing) in response to a predetermined load or throughput in the information processing device 1B. The change in data aggregation in the data aggregator 30B will be described in more detail later.

[0141] The primary inference unit 20B executes primary inference in the same manner as the primary inference unit 20. The secondary inference unit 40B executes secondary inference in the same manner as the secondary inference unit 40. Furthermore, at least one of the primary inference unit 20B and the secondary inference unit 40B switches the trained model used for inference in response to a predetermined load or throughput in the information processing device 1B. At least one of the primary inference unit 20B and the secondary inference unit 40B may switch between three or more trained models instead of two. When both the primary inference unit 20B and the secondary inference unit 40B switch the trained models, the primary inference unit 20B and the secondary inference unit 40B may use the same threshold value or different threshold values ​​as the threshold value used to determine the load or throughput.

[0142] The load and throughput used for the judgment in the second embodiment are arbitrary. An operator may determine the load or throughput used for the judgment based on the target object and the learned model, etc. For example, the information processing device 1B may use at least one of the following items as the load or throughput. The information processing device 1B may use a combination of two or more items instead of one item. When using the following values, the information processing device 1B may use statistical processing such as averaging. (1) Hardware resources related to computation constituting the information processing device 1B, (1-1) Utilization or operation rate of at least one of the CPU and ALU (GPU, FPGA, or ASIC), (1-2) The length of a data queue for processing in at least one of the components of the information processing device 1B, or the processing waiting time, (1-3) Measurement values ​​(e.g., temperature or power consumption) of sensors (not shown) included in the information processing device 1B, (2) The size of the data to be inferred, (3) A result of the primary inference in the primary inference unit 20B (e.g., the number or area of ​​BBs to be subjected to post-processing), (4) the result of post-processing in the data aggregation unit 30B (for example, the number or area of ​​BBs after post-processing); (5) the result of aggregation in data aggregator 30B (e.g., the size of the aggregated data); (6) The ratio of the size of the data to be inferred to the size of the aggregated data, (7) A result of the secondary inference in the secondary inference unit 40B (for example, the number or area of ​​BBs included in the secondary inference), (8) Processing time of at least one of the primary inference and the secondary inference (the processing time may be an actually measured time, or may be a time calculated using a predetermined relationship between the size of the data to be inferred and the processing time).

[0143] The information processing device 1B may switch the inference parameters based on other information instead of the load and throughput, or in addition to the load or throughput. For example, the shooting range of a surveillance camera of a system for detecting an intruder may include ranges with different required inference accuracy. For example, in a range near the entrance, it is desirable to have high inference accuracy. On the other hand, in a range far from the entrance, it is desirable to lower the load required for inference even if the inference accuracy is low in order to reduce unnecessary power consumption. Therefore, for example, the information processing device 1B usually uses inference parameters with low load (for example, a trained model with low load). Then, when a sensor (for example, a human sensor) detects a person's intrusion into a predetermined range (for example, approaching the vicinity of the entrance), the information processing device 1B may switch to inference parameters with high inference accuracy (for example, a trained model with high inference accuracy). In this way, the information processing device 1B may switch the inference parameters in response to predetermined information such as an event detected by the sensor.

[0144] The changes of the inference parameters in the primary inference unit 20B and the secondary inference unit 40B include the changes of the trained model in the above description, as follows. However, the changes of the inference parameters are not limited to the following. (1) Changing the trained model used for inference: When a trained model with a low load is used, the throughput of the primary inference unit 20B and the secondary inference unit 40B improves. However, the inference accuracy decreases. When a trained model with a high load but high accuracy is used, the throughput of the primary inference unit 20B and the secondary inference unit 40B decreases. However, the inference accuracy increases. (2) Changes in a given quantity that together perform first-order inference: When the predetermined number of times that the primary inferences are executed collectively is reduced, the information processing device 1 can shorten the time (latency) required to obtain a final inference result for the inference target data.

[0145] Next, a description will be given of changes to inference parameters (particularly data aggregation) related to the data aggregator 30B. The data aggregator 30B may change the following inference parameters based on a predetermined load or throughput in the information processing device 1B. (1) Changes in parameters for data aggregation (e.g., BB filtering or thresholds for BB adjustment or BB correction amount): When the filtering threshold is changed so that the number of filtering targets is reduced (so that the amount of data after filtering is reduced), the throughput of the data aggregation unit 30B improves. However, the inference accuracy of the secondary inference unit 40B decreases. When the filtering threshold is changed so that the number of filtering targets is increased (so that the amount of data after filtering is increased), the throughput of the data aggregation unit 30B decreases. However, the inference accuracy of the secondary inference unit 40B improves. (2) Changing the gap between duplicated data in the aggregate: When the gap is narrowed, the throughput of the data aggregating unit 30B is improved. However, the inference accuracy of the secondary inference unit 40B is reduced. When the gap is widened, the throughput of the data aggregating unit 30B is reduced. However, the inference accuracy of the secondary inference unit 40B is improved. (3) How to group areas: (3-1) Change whether to combine overlapping and adjacent areas: When the regions are not combined, the throughput of the data aggregating unit 30B is improved. However, the inference accuracy of the secondary inference unit 40B is reduced. When the regions are combined, the throughput of the data aggregating unit 30B is reduced. However, the inference accuracy of the secondary inference unit 40B is improved. (3-2) Changes to the distance that determines proximity: When the distance for determining proximity is shortened, the throughput of the data aggregating unit 30B improves. However, the inference accuracy of the secondary inference unit 40B decreases. When the distance for determining proximity is lengthened, the throughput of the data aggregating unit 30B decreases. However, the inference accuracy of the secondary inference unit 40B improves. (4) Change in size of aggregated data: When the size of the aggregated data is reduced, the throughput of the data aggregation unit 30B is improved. However, the inference accuracy of the secondary inference unit 40B is reduced. When the size of the aggregated data is increased, the throughput of the data aggregation unit 30B is reduced. However, the inference accuracy of the secondary inference unit 40B is improved.

[0146] The information processing device 1B may change items related to a plurality of components as follows as inference parameters. (1) Change in use of sub-objects: The information processing device 1B may switch whether or not to use a sub target object depending on the load. When the sub target object is not used, the throughput of the information processing device 1B is improved. However, the final inference accuracy is reduced. When the sub target object is used, the throughput of the information processing device 1B is reduced. However, the final inference accuracy is improved. (2) Presence or absence of first-order inference: The information processing device 1B may switch whether or not to use primary inference (primary inference unit 20B) depending on the load. When the amount of aggregated data is large and primary inference is not used, the throughput of the information processing device 1B improves. When the amount of aggregated data is large and primary inference is used, the throughput of the information processing device 1B decreases. In this way, the switching of primary inference included in the inference parameters includes whether or not to execute primary inference. Note that the operation of other components when primary inference is not used is optional. For example, when primary inference is not used, the data aggregation unit 30B may generate aggregated data using the inference target data. Alternatively, when primary inference is not used, the secondary inference unit 40B may infer a target object in the inference target data as secondary inference.

[0147] [Explanation of operation] Next, the operation of the information processing device 1B according to the second embodiment will be described with reference to the drawings. In the following description, the average value of the quantity of aggregated data generated by the data aggregation unit 30B per unit time (hereinafter referred to as "aggregated data generation speed") is used as an example of the load. In addition, the trained model used by the primary inference unit 20B (trained model for primary inference) is used as the inference parameter to be changed. The information processing device 1B operates so as to obtain higher inference accuracy in a low load situation. Specifically, when the aggregated data generation speed is equal to or lower than a threshold (when the load is low), the primary inference unit 20B uses a trained model with high accuracy (i.e., a trained model with high load) for the primary inference. Conversely, when the aggregated data generation speed exceeds the threshold (when the load is high), the primary inference unit 20B uses a trained model with low load (i.e., a trained model with low accuracy) for the primary inference.

[0148] FIG. 12 is a flow diagram showing an example of a model switching operation in the information processing device 1B according to the second embodiment. FIG. 12 is an example of an operation in which the primary inference unit 20B switches a trained model for primary inference used for primary inference in response to the aggregated data generation speed. The information processing device 1B calculates the aggregated data generation speed (step S201). The information processing device 1B determines whether the aggregated data generation speed exceeds a predetermined threshold (step S202). If the aggregated data generation speed exceeds the threshold (Yes in step S202), the primary inference unit 20B executes primary inference using a trained model with a low load as the primary inference (step S205). If the aggregated data generation speed is equal to or lower than the threshold (No in step S202), the primary inference unit 20B executes primary inference using a trained model with high accuracy as the primary inference (step S206). The information processing device 1B repeats the above operation at a predetermined interval. Using such an operation, the information processing device 1B switches the trained model for primary inference used by the primary inference unit 20B in response to the throughput (or load) of the data aggregating unit 30B.

[0149] [Effect description] The information processing device 1B according to the second embodiment can further improve the throughput in addition to the effect of the first embodiment. Alternatively, the information processing device 1B according to the second embodiment can improve the accuracy of inference in addition to the effect of the first embodiment. The reason is as follows.

[0150] The information processing device 1B switches at least one of the inference parameters (primary inference, secondary inference, and data aggregation) based on a predetermined load or throughput in the information processing device 1B. For example, the information processing device 1B switches at least one of the operations or data handled by the primary inference unit 20B, the secondary inference unit 40B, and the data aggregation unit 30B based on a predetermined load or throughput in the information processing device 1B.

[0151] For example, the information processing device 1B switches at least one of the trained model for primary inference used by the primary inference unit 20B and the trained model for secondary inference used by the secondary inference unit 40B based on the load or throughput of the data aggregating unit 30B. Alternatively, the information processing device 1B switches the data aggregation process in the data aggregating unit 30B based on the load or throughput of the primary inference unit 20B or the secondary inference unit 40B. In this way, the information processing device 1B according to the second embodiment changes inference parameters according to a predetermined load or throughput in the information processing device 1B.

[0152] By using such operations, the information processing device 1B suppresses a decrease in throughput and avoids the occurrence of data that cannot be processed when the load increases. On the other hand, when the load decreases, the information processing device 1B switches the inference parameters so that the load increases. By using such operations, the information processing device 1B realizes inference with appropriate accuracy while ensuring throughput.

[0153] <Third embodiment> The information processing devices 1 and 1B may acquire the trained model for primary inference and the trained model for secondary inference from other devices not shown in the figure. Alternatively, the information processing devices 1 and 1B may use inference target data stored in a predetermined storage device.

[0154] FIG. 13 is a block diagram showing an example of the configuration of an information processing device 1C according to the third embodiment. The information processing device 1C includes an object inference unit 10, a primary inference unit 20, a data aggregation unit 30, and a secondary inference unit 40. The primary inference unit 20 applies inference target data, at least a part of which includes a first target object, to a first learned model to infer the first target object as the primary inference. The data aggregation unit 30 uses the first target object inferred in the primary inference to generate aggregated data, which is data with a smaller amount than the inference target data. Furthermore, the data aggregation unit 30 generates a correspondence relationship between the position of the first target object in the inference target data and the position of the first target object in the aggregated data. The secondary inference unit 40 applies the aggregated data to a second learned model to infer the first target object as the secondary inference. The object inference unit 10 uses the first target object in the result of the secondary inference and the correspondence relationship to infer the first target object in the inference target data.

[0155] The information processing device 1C configured in this manner can obtain the same effect as the information processing device 1. That is, the information processing device 1C can achieve the effect of improving the throughput of object inference. Note that the information processing device 1C is the minimum configuration of the information processing device 1.

[0156] Although the present invention has been described above with reference to the embodiment, the present invention is not limited to the above embodiment. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Explanation of symbols]

[0157] 1. Information processing device 1B Information processing equipment 1C Information processing equipment 10 Object reasoning part 20 Primary reasoning part 20B Primary reasoning part 30 Data Aggregation Department 30B Data Aggregation Department 40 Secondary reasoning part 40B Secondary reasoning part 50 Data Storage Section 50B Data storage section 60 Data Generation Division 60B Data Generation Section 70 Model Generation Unit 70B Model Generation Section 80 Model Storage Section 80B Model Storage Section 90 Data Acquisition Section 200 Data Acquisition Device 300 display device 400 Information Processing Systems 610 CPU 611 ALU 620 ROM 630 RAM 640 Storage device 650NIC 690 Storage medium

Claims

1. A primary inference means for inferring the first target object by applying inference target data, at least a portion of which includes a first target object, to a first trained model as a primary inference; a data aggregation means for generating aggregated data, which is data having a smaller amount than the inference target data, by using the first object inferred in the primary inference, and for generating a correspondence relationship between a position of the first object in the inference target data and a position of the first object in the aggregated data; A secondary inference means for inferring the first target object by applying the aggregated data to a second trained model as a secondary inference; an object inference means for inferring the first object in the inference target data by using the first object in the result of the secondary inference and the correspondence relationship; An information processing device comprising:

2. The primary inference means infers a second object having a predetermined positional relationship with the first object, The data aggregating means generates the aggregated data and the correspondence relationship using the first target object and the second target object. The information processing device according to claim 1 .

3. The data aggregation means uses the second object included in the result of the primary inference, the first object being in the positional relationship with the second object included in the result of the primary inference, the second object being not included in the result of the primary inference. The information processing device according to claim 2 .

4. The data aggregating means performs a predetermined process on at least a part of the first target object included in the result of the primary inference before generating the aggregated data.

4. The information processing device according to claim 1.

5. A data generating means for generating a first training dataset used for training the first trained model using a first dataset, and generating a second training dataset used for training the second trained model using at least one of the first dataset and the first training dataset; a model generation means for generating the first trained model using the first training data set and generating the second trained model using the second training data set; The information processing apparatus according to claim 1 , further comprising:

6. Switching at least one of the primary inference, the secondary inference, and the data aggregation based on a predetermined load or throughput in the information processing device.

6. The information processing apparatus according to claim 1.

7. An information processing device according to any one of claims 1 to 6, a data acquisition device that outputs the inference target data to the information processing device; a display device that acquires an inference result from the information processing device and displays the acquired inference result; An information processing system comprising:

8. As a first inference, inference target data, at least a part of which includes a first target object, is applied to a first trained model to infer the first target object; generating aggregated data, which is data having a smaller quantity than the inference target data, using the first object inferred in the primary inference, and generating a correspondence relationship between a position of the first object in the inference target data and a position of the first object in the aggregated data; As a secondary inference, applying the aggregated data to a second trained model to infer the first target object; Inferring the first object in the inference target data using the first object in the result of the secondary inference and the correspondence relationship. Information processing methods.

9. An information processing device executes the information processing method according to claim 8, a data acquisition device outputs the inference target data to the information processing device; A display device acquires the result of the inference from the information processing device and displays the acquired result of the inference. Information processing methods.

10. As a first inference, a process of applying inference target data, at least a part of which includes a first target object, to a first trained model to infer the first target object; a process of generating aggregated data, which is data having a smaller quantity than the inference target data, by using the first target object inferred in the primary inference; and a process of generating a correspondence relationship between a position of the first target object in the inference target data and a position of the first target object in the aggregated data; A process of inferring the first target object by applying the aggregated data to a second trained model as a secondary inference; a process of inferring the first target object in the inference target data by using the first target object in the result of the secondary inference and the correspondence relationship; A program that causes a computer to execute the following.

Images