Model training device, model training method and program

Abstract

An object of the present disclosure is to provide a model training apparatus, a model training method, and a non-transitory computer-readable medium that can provide a time-efficient method for determining optimal Feature Pyramid Network (FPN) count parameter values ​​for an Artificial Convolutional Neural Network (ACNN) model. In one aspect, the model training apparatus (100) includes an estimation unit (101) configured to estimate values ​​of FPN blocks included in the ACNN model using a training dataset, and an ACNN model training unit (102) configured to train the ACNN model using the estimated values.

Description

[Technical field]

[0001] The present disclosure relates to a model training apparatus, a model training method, and a non-transitory computer-readable medium. [Background technology]

[0002] Over the past decade, Artificial Convolutional Neural Network (ACNN) methods have become popular and have become state-of-the-art solutions for several vision processing applications such as classification, object detection, and semantic segmentation.

[0003] As an example of related technology, Patent Document 1 (PTL1) discloses a learning method for improving the performance of a convolutional neural network (CNN) using a feature up-sampling network. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] European Patent Application Publication No. 3467720 Summary of the Invention [Problem to be solved by the invention]

[0005] In the field of ACNN, feature pyramid network (FPN) technology has been recently developed. To achieve high accuracy, more FPN blocks in an ACNN model are needed. However, the more FPN blocks an ACNN model contains, the longer it takes to run. Therefore, there is a trade-off between accuracy and run time.

[0006] An objective of the present disclosure is to provide a model training apparatus, a model training method, and a non-transitory computer-readable medium that can provide a time-efficient method for determining optimal FPN count parameter values ​​for an ACNN model. [Means for solving the problem]

[0007] According to a first aspect of the present disclosure, there is provided a model training device including an estimation means for estimating numerical values ​​of Feature Pyramid Network (FPN) blocks included in an Artificial Convolutional Neural Network (ACNN) model using a training dataset, and an ACNN model training means for training the ACNN model using the estimated values.

[0008] According to a second aspect of the present disclosure, there is provided a model training method for estimating numerical values ​​of FPN blocks included in an ACNN model using a training dataset, and training the ACNN model using the estimated values.

[0009] According to a third aspect of the present disclosure, there is provided a non-transitory computer-readable medium having stored thereon a program for causing a computer to estimate numerical values ​​of FPN blocks included in an ACNN model using a training dataset, and to train the ACNN model using the estimated values. Effect of the Invention

[0010] According to the present disclosure, it is possible to provide a model training apparatus, a model training method, and a non-transitory computer-readable medium that can provide a time-efficient method for determining optimal FPN count parameter values ​​for an ACNN model. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is an example of a block diagram of an ACNN model according to the related art. [Diagram 2] FIG. 2 is an example of an MLFPN block according to the related art. [Diagram 3] FIG. 3 is an example of an FPN block according to the related art. [Figure 4] FIG. 4 is an example of a block diagram of the model training device according to the first embodiment. [Diagram 5] FIG. 5 is a flowchart showing an example of a method of the model training device according to the first embodiment. [Figure 6] FIG. 6 is an example of a block diagram of an ACNN model training system according to the second embodiment. [Figure 7] FIG. 7 is an example of a block diagram of the count estimation unit according to the second embodiment. [Figure 8] FIG. 8 is an example of a table showing the estimation results based on the training data set TD. [Figure 9] FIG. 9 is another example of a table showing the estimation results based on the training data set TD. [Figure 10] FIG. 10 is a flowchart illustrating an example of all the processes of the ACNN model training system according to the second embodiment. [Figure 11] FIG. 11 is a flowchart illustrating an example of the process of the count estimation unit according to the second embodiment. [Figure 12] FIG. 12 is a flowchart illustrating an example of a process performed by the feature point extraction unit according to the second embodiment. [Figure 13] FIG. 13 shows an example of feature points of an image. [Figure 14] FIG. 14 is an example of Scale Invariant Feature Transform (SIFT) feature point extraction in a particular image. [Figure 15] FIG. 15 is an example of a block diagram of a computer device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] (Summary of Related Technology) Before describing the embodiments of the present disclosure, an overview of related technologies will be described. The following related technologies are disclosed in “Feature Pyramid Networks for Object Detection”, Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, Serge Belongie; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2117-2125, and are incorporated herein by reference as part of this document.

[0013] FIG. 1 shows an ACNN model 10 (hereinafter, referred to as ACNN 10). As shown in FIG. 1, ACNN 10 includes several important blocks, specifically, a backbone block 11, a fusion block 12, a multi-level feature pyramid network (MLFPN) block 13, and a head block 14. ACNN 10 receives an image IM as input, processes it, and outputs a prediction PR based on the image IM. The architecture of each block of ACNN 10 is designed with the aim of achieving high accuracy for a given dataset. In addition, for real-time applications, the architecture of ACNN 10 is designed with the additional aim of achieving a desired execution speed.

[0014] The backbone block 11 functions as a backbone convolutional architecture, and the fusion block 12 integrates data inputs from the backbone block 11. The head block 14 is the convolutional layer of the Residual Network (ResNet).

[0015] The MLFPN block 13 is one of the main components of the ACNN 10. As shown in FIG. 2, the MLFPN block 13 is formed by stacking multi-level feature pyramid network (FPN) blocks 15, and is known as a "multi-level feature pyramid network block."

[0016] The FPN block 15 is a feature extraction block designed to detect objects of various scales in the input image. A sample example of the FPN block 15 is shown in FIG. 3. FIG. 3 shows the architecture of the FPN block 15, which includes layers L1, L2, L3, ..., L4, L5, ..., L6 and L7. Each of the layers L1-L4 in the bottom-up step generates feature maps of different scales, and each of the layers L5-L7 in the top-down step outputs a prediction based on the feature map from the corresponding layer in the bottom-up step. For example, layer L5 outputs a prediction P1 using the feature map from layer L4, layer L6 outputs a prediction P2 using the feature map from layer L3, and layer L7 outputs a prediction P3 using the feature map from layer L2. Conventionally, the architecture of one FPN block 15 is optimized by experts, and then such optimized FPN blocks 15A, 15B, ... 15C are stacked to form the MLFPN block 13.

[0017] Having multiple levels of FPN blocks 15 in the MLFPN block 13 is necessary to achieve high accuracy by the ACNN 10. However, increasing the levels of FPN blocks 15 in the MLFPN block 13 results in large computational complexity of the ACNN 10, thereby increasing the execution time of the ACNN 10. The long execution time is not acceptable for real-time applications. Also, having one or a few levels of FPN blocks 15 in the MLFPN block 13 to enjoy lower computational complexity and lower execution time may result in low accuracy. Therefore, the number of FPN blocks 15 in the MLFPN block 13 is a key parameter of the ACNN 10, and is referred to as the FPN count value in this document. Determining the optimal value of the FPN count value, considering the trade-off between accuracy and computational complexity or execution time for a given real-time application and dataset, is an optimization problem.

[0018] A simple solution to the optimization problem is to exhaustively enumerate all candidate values ​​of the FPN count. However, the total time required to enumerate all candidate values ​​is the product of the number of candidate values ​​and the time to train each ACNN model for a given dataset. Therefore, since the FPN count parameter can have multiple positive integers as candidate options, the total time required to exhaustively enumerate can be very large.

[0019] Another solution to this problem is to use machine learning algorithms such as reinforcement learning and genetic algorithms to solve the optimization problem. These machine learning algorithms basically explore a small number of candidate values ​​and predict the optimal FPN count value. However, even the time required to explore a small number of candidate values ​​is long.

[0020] In view of this related art, one objective of this disclosure is to provide a time-efficient method for determining an optimal FPN count parameter value for an MLFPN block in an ACNN.

[0021] (Embodiment 1) First, with reference to FIG. 4, the model training device 100 according to the first embodiment of the present disclosure will be described.

[0022] 4, the model training apparatus 100 includes an estimation unit 101 and an ACNN model training unit 102. The model training apparatus 100 includes, for example, one or more computers or machines. As an example, at least one component of the model training apparatus 100 can be mounted on a computer as a combination of one or more memories and one or more processors.

[0023] The estimation unit 101 estimates the numerical values ​​of the FPN blocks included in the ACNN model using a training data set (not shown in FIG. 4). The ACNN model training unit 102 trains the ACNN model using the values ​​output by the estimation unit 101. The FPN blocks can be stacked in (but are not limited to) an MLFPN structure, a single-shot multi-box detector (SSD) type feature pyramid structure, or an FPN type feature pyramid structure. For example, the structure of the ACNN model that constitutes the MLFPN block is shown in FIGS. 1 and 2, and this structure is described in the "Overview of Related Art". Therefore, a detailed description thereof will be omitted here.

[0024] Next, an example of the operation of this embodiment will be described with reference to the flowchart of FIG.

[0025] First, the estimation unit 101 estimates the numerical values ​​of the FPN blocks included in the ACNN model using a training data set (step S11 in FIG. 5). Next, the ACNN model training unit 102 trains the ACNN model using the estimated values ​​(step S12 in FIG. 5).

[0026] Since the estimation unit 101 estimates a value indicating the number of FPN blocks required for the ACNN model, the ACNN model training unit 102 trains the ACNN model so that the number of FPN blocks is the minimum required, thus providing a time-efficient method for determining an optimal FPN count parameter value for the ACNN model.

[0027] (Embodiment 2) Next, a second embodiment of the present disclosure will be described with reference to the accompanying drawings. The second embodiment shows a specific example of the first embodiment, but the specific example of the first embodiment is not limited to this.

[0028] Fig. 6 shows an ACNN model training system (hereinafter, referred to as a model training system) according to the second embodiment. As shown in Fig. 6, the model training system 200 includes a count estimation unit 201 and an ACNN model training unit (hereinafter, referred to as a model training unit) 202. The model training system 200 takes in a training dataset TD as an input and outputs a trained ACNN model.

[0029] The model training system 200 can be realized as a system mounted on an information processing device. The model training system 200 may include other units for calculation. In this embodiment, the model training system 200 can be applied to image recognition, but the application is not limited thereto.

[0030] The training dataset TD contains images with labels and ground truth bounding box information for each image. The labels are the predictions that the model training system 200 tries to obtain as the output of the ACNN. The ground truth bounding box information is used for cropping by the feature point extraction unit 211.

[0031] The count estimation unit 201 takes the training data set TD as input, performs a complexity analysis of the training data set TD, and estimates and outputs the value of the FPN count parameter.

[0032] Figure 7 shows details of the count estimation unit 201. As shown in Figure 7, the count estimation unit 201 includes a feature point extraction unit 211 and an estimation unit 212. The feature point extraction unit 211 takes the training dataset TD as input and generates filtered feature points for all the training dataset images as output.

[0033] The estimator 212 takes the filtered feature points as input and performs feature point analysis to provide the value of the FPN count parameter as output. In the feature point analysis, the estimator 212 adds up the number of objects of each target class in the training data set TD and adds up the number of feature points of all objects in each class.

[0034] Fig. 8 shows an example of a table showing the estimation results based on the training dataset TD. In the table in Fig. 8, column C0 shows the class ID of each target class, column C1 shows a sample example of the number of objects in each target class, and column C2 shows a sample example of the total number of feature points in each target class.

[0035] The estimation unit 212 uses information on the number of objects in each target class and the total number of feature points in each target class (information in columns C1 and C2) to calculate an average feature point (AFP) value as shown in column C3 of the table in FIG. 8. The AFP value can be calculated by dividing the total number of feature points by the number of objects. The estimation unit 212 then uses the AFP value information to estimate the complexity of each target class as shown in column C4 of the table in FIG. 8. The higher the AFP value, the higher the complexity. The estimation unit 212 then calculates count information for each target class based on the complexity as shown in column C5 of the table in FIG. 8. The higher the complexity, the higher the count value.

[0036] Fig. 9 shows another example of a table showing the estimation results based on the training data set TD. In the table of Fig. 9, columns C10 to C15 correspond to columns C0 to C5, respectively.

[0037] Figures 8 and 9 show that object classes with low AFP values ​​can be considered less complex and assigned a smaller count value, while object classes with high AFP values ​​can be considered more complex and assigned a larger count value.

[0038] For low-complexity datasets, a low-complexity deep CNN model is needed. In other words, a small number of FPN blocks in the MLFPN part is enough to achieve good accuracy on low-complexity datasets. On the other hand, for high-complexity datasets, a high-complexity deep CNN model is needed. In other words, a large number of FPN blocks in the MLFPN part is needed to achieve good accuracy on high-complexity datasets. Therefore, by using the number of AFP values ​​as the complexity analysis metric, the FPN count value of each target class can be estimated.

[0039] The estimation unit 212 outputs the maximum value of column C5 or C15 as the FPN count parameter value. The FPN count parameter value is also the output of the count estimation unit 201. In the case of FIG. 8, the maximum value of column C5 is the value "4", so the estimation unit 212 outputs the value "4". In the case of FIG. 9, the maximum value of column C15 is the value "8", so the estimation unit 212 outputs the value "8".

[0040] Referring again to Fig. 6, the model training unit 202 uses the FPN count parameter value and the training dataset TD as input to train the ACNN model, and outputs the trained ACNN model TM as shown in Fig. 6. The ACNN model trained by the model training unit 202 is stored in the model training system 200, the structure of which is shown in Figs. 1 and 2. The description of this structure has already been given in the "Overview of Related Art", so it will not be explained here.

[0041] Next, an example of the overall processing of the model training system 200 will be described with reference to the flowchart of FIG.

[0042] First, the count estimation unit 201 estimates and outputs the FPN count parameter value from the training data set TD (step S21 in FIG. 10). Next, the model training unit 202 uses the FPN count parameter value and the training data set TD as inputs to train the ACNN model and outputs the trained ACNN model TM (step S22 in FIG. 10).

[0043] Further, an example of the processing of the count estimation unit 201 will be described with reference to the flowchart of Fig. 11. That is, details of the processing of step S21 will be described below.

[0044] First, the feature point extraction unit 211 takes the training data set TD as input and generates filtered feature points for all training data set images as output (step S31 in FIG. 11). Next, the estimation unit 212 analyzes the feature points and estimates the FPN count parameter value (step S32 in FIG. 11).

[0045] Further, an example of the process of the feature point extraction unit 211 will be described with reference to the flowchart of Fig. 12. That is, details of the process of step S31 will be described below.

[0046] First, the feature point extraction unit 211 uses the ground truth bounding box information of the training dataset TD to crop all objects as individual images (step S41 in FIG. 12). This cropping is performed to avoid future analysis of the background and to isolate all objects (all images) in the training dataset TD. The future analysis refers to the processing in steps S43 to S45 and S32. Background regions without target objects may not be subject to this future analysis in terms of time cost. However, if the training dataset TD contains one object per image and the background is minimal, step S41 is optional and can be skipped. The output of this cropping process is individual images of all target objects in the training dataset TD. In the case of FIG. 8, the feature point extraction unit 211 generates information of column C1 in step S45. In the case of FIG. 9, the feature point extraction unit 211 generates information of column C11 in step S45.

[0047] Second, the feature point extraction unit 211 pre-processes all input images and outputs pre-processed images (step S42 in FIG. 12). Pre-processing includes, but is not limited to, resizing images with different resolutions. This image pre-processing is performed to normalize all objects of different scales and classes. However, if all objects are of the same scale in all classes in the training dataset TD, step S42 is optional and can therefore be skipped.

[0048] Third, the feature point extraction unit 211 performs feature point extraction on all the processed images (step S43 in FIG. 12). A feature point is a spatial location or point in an image that defines something interesting or salient in the image. Another name for a feature point can be a key point or a point of interest in an image. There may be more than one such feature point in an image, so there will be one or more extracted feature points by the extraction.

[0049] Figure 13 shows a sample example of feature points (keypoints) in an image. As shown in Figure 13, the image contains an object (a rectangle), and the object has some edges, and these edges define the shape of the rectangle. Feature points 1 and 2 in Figure 13 are examples of these edges. We can also extract the edges of the rectangle as feature points.

[0050] There are several well-known feature extraction techniques, such as Scale Invariant Feature Transform (SIFT), Speeded up robust features (SURF), Binary Robust Invariant Scalable Feature Points (BRISF), etc. Figure 14 shows a sample example of SIFT feature extraction applied to a particular image. The image contains a portrait of a person. The feature points are represented by small circles at the points of interest of the particular image in Figure 14. The points of interest in Figure 14 can be used to detect features, especially people. The number of extracted feature points varies depending on the image to which the feature extraction technique is applied. Generally, if the number of extracted feature points is low in a particular image, the particular image is called a simple image. On the other hand, if the number of extracted feature points is high in a particular image, the particular image is called a complex image.

[0051] Referring again to FIG. 12, after step S43, the feature point extraction unit 211 filters the extracted feature points of each image appropriately (step S44 in FIG. 12). This filtering is an optional step, but may be necessary if some of the extracted feature points belong to the background. Another case in which filtering may be necessary is to remove certain feature points that belong to a particular category. For example, if certain feature points are too small or too large, they will be removed.

[0052] Finally, the feature point extraction unit 211 outputs the filtered feature points for all images (step S45 in FIG. 12). In the case of FIG. 8, in step S45, the feature point extraction unit 211 outputs information of column C2. In the case of FIG. 9, in step S45, the feature point extraction unit 211 outputs information of column C12. Thereafter, the estimation unit 212 estimates the FPN count parameter value using the filtered feature points. In the case of FIG. 8, the estimation unit 212 calculates the parameters of columns C3 to C5. In the case of FIG. 9, the estimation unit 212 calculates the parameters of columns C13 to C15. The details of this calculation are as described above.

[0053] It should be noted that this feature extraction method needs to be applied to the training dataset TD only once. Moreover, the computation in the feature extraction unit 211 of the count estimation unit 201 is not very complex, and there is no need to repeat the training process of the ACNN model. As a result, the proposed method is faster than other existing methods.

[0054] As explained above, one of the problems in the related art is that it is difficult to search for several candidate values ​​for the FPN count architecture parameter in the MLFPN block of ACNN. The reason for this problem is that there are several candidate values ​​to be searched for the FPN count architecture parameter. Also, there is a trade-off between computational complexity and accuracy.

[0055] The count estimation unit 201 estimates a value indicating the number of FPN blocks required for the ACNN model, and therefore provides a time-efficient method for determining an optimal FPN count parameter value in the ACNN model. In addition, in this embodiment, the ACNN model includes an MLFPN block having an FPN block, and the count estimation unit 201 estimates the number of FPN blocks stacked in the MLFPN block. Thus, the ACNN model training system 200 can train an ACNN model including an MLFPN block.

[0056] Furthermore, the estimation unit 212 estimates the value of the FPN block based on the number of objects and feature points of each target class in the training dataset TD, which allows the ACNN model training unit 202 to set the level of the FPN block corresponding to the content of the image.

[0057] In addition, the estimation unit 212 calculates each numerical value of the FPN block of each target class, and outputs the maximum value of the FPN block number of each target class as the estimated value. Therefore, the ACNN model training unit 202 can set the level of the FPN block as the minimum value, and the parameters of the FPN block ensure that the calculation result by the ACNN model is correct.

[0058] In addition, the feature point extraction unit 211 extracts feature points of the images in the training data set TD and estimates the numerical values ​​of the FPN blocks using the feature points, so that the ACNN model training unit 202 can set the level of the FPN blocks to make the ACNN model suitable for object identification.

[0059] Furthermore, the feature point extraction unit 211 filters the extracted feature points, and the estimation unit 212 estimates the numerical value of the FPN block using the filtered feature points. The count estimation unit 201 can set the smallest possible value, so that the level of the FPN block can be reduced.

[0060] Next, a configuration example of the information processing device described in the above embodiments will be described below with reference to FIG.

[0061] 15 is a block diagram showing an example of the configuration of an information processing device 90. As shown in FIG.

[0062] The network interface 91 is used for communication with other network node devices constituting the communication system. For example, the network interface 91 may receive the training data set TD.

[0063] The processor 92 loads software (computer programs) from the memory 93 and executes the loaded software to perform the processes described with reference to the drawings of the above embodiment. The processor 92 may be, for example, a microprocessor, a micro processing unit (MPU), or a central processing unit (CPU). The processor 92 may include multiple processors. In this case, each processor executes one or more programs including a group of instructions to cause the computer to execute the algorithms described above with reference to the drawings.

[0064] The memory 93 may be composed of a volatile memory or a nonvolatile memory, or may be composed of a combination of a volatile memory and a nonvolatile memory. The memory 93 may include a storage device disposed remotely from the processor 92. In this case, the processor 92 may access the memory 93 via an I / O interface (not shown).

[0065] 15, a memory 93 is used to store software modules. The processor 92 can perform the processes described in the above embodiments by reading the software modules from the memory 93 and executing the read software modules.

[0066] As described above, each configuration in the above embodiments may be configured by a combination of hardware and software (computer programs). However, each configuration may be configured by a single piece of hardware or software, or may be configured by multiple pieces of hardware or software.

[0067] The computer program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions described in the embodiments. The computer program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the non-transitory computer-readable medium or the tangible storage medium may be a random-access memory (RAM), a read-only memory (ROM), a flash memory, a solid-state drive (SSD) or other memory technology, a CD-ROM, a digital versatile disk (DVD), a Blu-ray (Registered Trademark) The medium may include a disk or other optical disk storage, a magnetic cassette, a magnetic tape, a magnetic disk storage or other magnetic storage device. The computer program may be transmitted on a transitory computer readable medium or a communication medium. By way of example and not limitation, the transitory computer readable medium or communication medium may include an electrical, optical, acoustic, or other form of propagated signal.

[0068] The present disclosure is not limited to the above-described embodiments, and can be modified as appropriate without departing from the spirit and scope of the present disclosure. [Explanation of symbols]

[0069] 100 Model Training Device 101 Estimation part 102 Artificial Convolutional Neural Network (ACNN) model training section 200 ACNN model training system 201 Count Estimation Department 202 ACNN Model Training Department 211 Feature point extraction unit 212 Estimation Department

Claims

1. An estimation means for obtaining a training dataset including a plurality of images having objects, deriving the number of objects and the number of feature points of all objects for each object class of the training dataset, calculating an Average Feature Point (AFP) value for each object class using the number of objects and the number of feature points, and estimating the number of Feature Pyramid Network (FPN) blocks corresponding to the maximum value of the AFP value for each object class as an optimal number of FPN blocks to be included in an Artificial Convolutional Neural Network (ACNN) model; An ACNN model training means for training the ACNN model using the estimated optimal number; A model training device comprising:

2. The estimation means calculates the AFP value for each target class by dividing the number of feature points by the number of objects.

2. The model training apparatus of claim 1.

3. The ACNN model includes a multi-level feature pyramid network (MLFPN) block that includes the FPN block, The estimation means estimates an optimal number of the FPN blocks stacked in the MLFPN block as the optimal number.

3. A model training device according to claim 1 or 2.

4. The estimation means extracts feature points for each image of the training dataset, performs filtering to remove feature points belonging to a specific category from the extracted feature points, and calculates the AFP value for each target class using the number of objects and the number of feature points on which filtering has been performed. A model training device according to any one of claims 1 to 3.

5. Obtaining a training dataset including a plurality of images having objects, deriving the number of objects and the number of feature points of all objects for each object class of the training dataset, calculating an Average Feature Point (AFP) value for each object class using the number of objects and the number of feature points, and estimating the number of Feature Pyramid Network (FPN) blocks corresponding to the maximum value of the AFP value for each object class as an optimal number of FPN blocks to be included in an Artificial Convolutional Neural Network (ACNN) model; training the ACNN model using the estimated optimal number; A computer implemented method for training a model.

6. Obtaining a training dataset including a plurality of images having objects, deriving the number of objects and the number of feature points of all objects for each object class of the training dataset, calculating an Average Feature Point (AFP) value for each object class using the number of objects and the number of feature points, and estimating the number of Feature Pyramid Network (FPN) blocks corresponding to the maximum value of the AFP value for each object class as an optimal number of FPN blocks to be included in an Artificial Convolutional Neural Network (ACNN) model; training the ACNN model using the estimated optimal number; A program that causes a computer to do something.

Images