IMAGE DATASET GENERATION DEVICE, LEARNING DEVICE, LEARNING SYSTEM, AND IMAGE DATASET GENERATION METHOD

Abstract

To suppress increase of learning time and overlearning.SOLUTION: An image dataset generation apparatus includes a generation unit and an exclusion unit. The generation unit generates an image dataset formed by cutting out an area corresponding to an object detection range from an image group and labeling the area with a class name. The exclusion unit excludes at least one piece of image data having similar feature quantities from the image dataset.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The disclosed embodiments include an image dataset generation device, a learning device, study A system and method for generating an image dataset. [Background technology]

[0002] Conventionally, image recognition technology is known that classifies images, detects objects contained in images, etc., using trained models such as DNNs (Deep Neural Networks) trained using machine learning algorithms. In addition, a method for generating a data set consisting of a large amount of image data, which is necessary as training data when using such technology, has also been proposed (for example, see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-029021 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, there is room for further improvement in the conventional techniques in terms of preventing an increase in learning time and overlearning.

[0005] For example, the above-mentioned dataset may contain redundant data, which is data with similar features. Although such redundant data has little contribution to improving image recognition performance, it may cause an increase in learning time and overlearning. For this reason, it is desirable to exclude redundant data from the dataset.

[0006] However, if one were to manually identify redundant data in order to remove it, manual checking would be costly in terms of manpower and time, and it would be difficult to accurately identify redundant data because the visual similarity in two-dimensional image space does not match the similarity in the model's feature space.

[0007] One aspect of the embodiments has been made in consideration of the above, and aims to provide an image dataset generation device, a learning device, an in-vehicle system, and an image dataset generation method that can prevent an increase in learning time and overlearning. [Means for solving the problem]

[0008] According to one aspect of the embodiment, an image dataset generating apparatus includes: controller The above-mentioned controller Is, Statue? Then, the area corresponding to the object detection range is extracted. The method generates training image data, generates a training image dataset consisting of the generated training image data, inputs the training image data in the training image dataset into a trained model to obtain features of each of the training image data, performs a cluster analysis based on the features of each of the training image data, calculates the distance from the center of gravity of the cluster to which each of the training image data belongs based on the results of the cluster analysis, sets an exclusion priority for the training image data belonging to the same cluster in order of closest distance from the center of gravity of the cluster, and creates a training image dataset by excluding the training image data based on the exclusion priority. Effect of the Invention

[0009] According to one aspect of the embodiment, it is possible to prevent an increase in learning time and overlearning. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic explanatory diagram of an image dataset generating method according to an embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating an example of the configuration of a generating device according to the embodiment. [Diagram 3] FIG. 3 is an explanatory diagram of the generation process executed by the generation unit. [Figure 4] FIG. 4 is an explanatory diagram of the feature acquisition process executed by the feature acquisition unit. [Diagram 5] FIG. 5 is an explanatory diagram of the cluster analysis process executed by the cluster analysis unit. [Figure 6] FIG. 6 is an explanatory diagram (part 1) of the determination process executed by the determination unit. [Figure 7]FIG. 7 is an explanatory diagram (part 2) of the determination process executed by the determination unit. [Figure 8] FIG. 8 is an explanatory diagram (part 3) of the determination process executed by the determination unit. [Figure 9] FIG. 9 is a flowchart illustrating a processing procedure executed by the generating device according to the embodiment. [Figure 10] FIG. 10 is a block diagram illustrating an example of the configuration of a learning system according to the embodiment. [Figure 11] FIG. 11 is a diagram (part 1) showing an operation example of the data collection system. [Figure 12] FIG. 12 is a diagram (part 2) showing an operation example of the data collection system. [Figure 13] FIG. 13 is a diagram (part 3) showing an operation example of the data collection system. [Figure 14] FIG. 14 is a diagram illustrating an example of tag data. [Figure 15] FIG. 15 is a block diagram showing an example of the configuration of a learning system according to the first modified example. [Figure 16] FIG. 16 is a block diagram showing an example of the configuration of a learning system according to the second modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of an image dataset generation device, a learning device, an in-vehicle system, and an image dataset generation method disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the embodiments described below.

[0012] In the following, an example will be described in which the generating device 10 according to the embodiment (corresponding to an example of an "image dataset generating device", see FIG. 2) is mounted on an in-vehicle device or the like and generates an image dataset for learning an object detection model that detects objects from images captured by a camera.

[0013] 1 is a schematic diagram for explaining an image dataset generation method according to an embodiment. In the image dataset generation method according to the embodiment, an image dataset is generated by extracting an area corresponding to an object detection range from a group of images and labeling the area with a class name, and at least one of image data having similar features is excluded from the image dataset.

[0014] 1, in the image dataset generation method according to the embodiment, the generation device 10 first generates an image dataset 11c, which is a group of image data obtained by cutting out bounding box regions B1-B5... corresponding to the detection range of each object from an "image group" that is the basis of the dataset (step S1). At this time, the generation device 10 labels each cut-out region with a class name.

[0015] Then, the generating device 10 trains the image classification model 11d by a machine learning algorithm using the image data set 11c (step S2). The machine learning algorithm is, for example, deep learning, and in this case, the image classification model 11d is a DNN.

[0016] Then, the generating device 10 uses the image classification model 11d trained in step S2 to infer all image data included in the image data set 11c, and acquires features for all the image data (step S3). The features are, for example, feature vectors in the feature space of the image classification model 11d.

[0017] Then, the generating device 10 executes a cluster analysis based on the feature information 11e, which is the feature acquired in step S3 (step S4). The cluster analysis is, for example, hierarchical clustering. In the image dataset generating method according to the embodiment, the generating device 10 executes hierarchical clustering based on the distance of the feature vectors.

[0018] The "distance" referred to here is Euclidean distance, cosine distance, Chebyshev distance, Manhattan distance, Tanimoto distance, etc. This allows image data having similar features to be classified into the same cluster.

[0019] The generating device 10 then identifies redundant data for all clusters based on the cluster information 11f resulting from the cluster analysis in step S4 (step S5).The generating device 10 then removes redundant data from the image dataset 11c as necessary (step S6) to generate a final dataset for training an object detection model (hereinafter, appropriately referred to as "final dataset 11g").

[0020] By training the object detection model using such final data set 11g, it is possible to prevent an increase in training time and overtraining.

[0021] In this way, in the image dataset generation method according to the embodiment, bounding box regions B1-B5... corresponding to the detection range of an object are extracted from a group of images, and an image dataset 11c is generated by labeling each region with a class name, and at least one of the image data having similar features is excluded from the image dataset 11c. In other words, the image dataset generation method according to the embodiment extracts a region corresponding to the detection range of an object from a group of images, and generates an image dataset by excluding at least one of the image data having similar features of the region.

[0022] Therefore, according to the image dataset generation method according to the embodiment, it is possible to prevent an increase in learning time and overlearning.

[0023] An example of the configuration of the generation device 10 to which the image dataset generation method according to the embodiment is applied will be described in more detail below.

[0024] Fig. 2 is a block diagram showing an example of the configuration of a generating device 10 according to an embodiment. Note that Fig. 2 and Figs. 10, 15, and 16 shown later show only components necessary for explaining the features of this embodiment, and descriptions of general components are omitted.

[0025] In other words, each component shown in Figure 2 and Figures 10, 15, and 16 is a functional concept, and does not necessarily have to be physically configured as shown. For example, the specific form of distribution and integration of each block is not limited to that shown in the figures, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0026] In addition, in the description using FIG. 2 and FIGS. 10, 15, and 16, the description of components that have already been described may be simplified or omitted.

[0027] As shown in Fig. 2, a generation device 10 according to the embodiment includes a storage unit 11 and a control unit 12. The storage unit 11 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk, and in the example of Fig. 2, stores an image DB (Data Base) 11a, generation parameter information 11b, an image dataset 11c, an image classification model 11d, feature amount information 11e, cluster information 11f, and a final dataset 11g.

[0028] The image DB 11a is a database that stores an image group 3 collected by a collection unit 12a (described later). The generation parameter information 11b is parameter information referenced by a generation unit 12b (described later), and includes, for example, the position and size of the bounding box region described above, and the class name to be labeled for each cut-out region.

[0029] The parameter information corresponding to the generation parameter information 11b may be appropriately specified by an operator or the like via an input device (not shown). The image data set 11c, the image classification model 11d, the feature amount information 11e, the cluster information 11f, and the final data set 11g have already been described, and therefore will not be described here.

[0030] The control unit 12 is a controller, and is realized, for example, by a CPU (Central Processing Unit), MPU (Micro Processing Unit), etc., executing various programs stored in a storage device inside the generating device 10 using a RAM as a working area. The control unit 12 can also be realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0031] The control unit 12 has a collection unit 12a, a generation unit 12b, a learning unit 12c, a feature acquisition unit 12d, a cluster analysis unit 12e, a judgment unit 12f, an exclusion unit 12g, and an output unit 12h, and realizes or executes the functions and actions of information processing described below.

[0032] The collecting unit 12a collects the image group 3 and stores it in the image DB 11a. Although not shown in the figure, the collecting unit 12a may collect the image group 3 from other devices via a network such as the Internet or a mobile phone network.

[0033] The generating unit 12b generates an image data set 11c based on each image data stored in the image DB 11a and the generation parameter information 11b. Here, Fig. 3 is an explanatory diagram of the generation process executed by the generating unit 12b.

[0034] As shown in FIG. 3, the generation unit 12b extracts a bounding box region B from each image P stored in the image DB 11a based on the generation parameter information 11b, labels each extracted region with a class name ("car" in the example of FIG. 3), and stores the labeled region in the image dataset 11c.

[0035] In terms of efficiency, it is preferable to automatically create the bounding box using a known processing method such as creating it from the detection results of an object detection model, but it is also possible to create the bounding box manually, for example by displaying the image to be processed on a display and having an operator give instructions using an input device such as a mouse or touch pad while visually checking the image. Subsequent processing is performed on the area of ​​the bounding box created in this way.

[0036] Returning to the description of Fig. 2, the learning unit 12c executes machine learning for image classification using the image dataset 11c generated by the generation unit 12b, and learns an image classification model 11d.

[0037] The feature acquisition unit 12d infers all image data included in the image dataset 11c using the image classification model 11d, which is a trained model trained by the learning unit 12c, acquires features for all such image data, and stores them in the feature information 11e.

[0038] Here, Fig. 4 is an explanatory diagram of the feature acquisition process executed by the feature acquisition unit 12d. As shown in Fig. 4, the image classification model 11d is a model that calculates a likelihood score for each label for an object appearing in image data, and outputs the label with the maximum score. In the example of Fig. 4, the image classification model 11d outputs the label "car" for the input image data.

[0039] As shown in Fig. 4, the image classification model 11d has a feature extraction unit 11da and a detection unit 11db. In terms of DNN, the feature extraction unit 11da corresponds to each layer before the fully connected layer. The detection unit 11db corresponds to each layer after the fully connected layer.

[0040] Then, the feature amount acquiring unit 12d acquires the feature amount vector of each image data extracted by the feature extracting unit 11da for each image data input to the image classification model 11d.

[0041] Returning to the description of Fig. 2, the cluster analysis unit 12e executes cluster analysis processing based on the feature amount information 11e acquired by the feature amount acquisition unit 12d, and stores the result in the cluster information 11f. As described above, the cluster analysis processing according to the embodiment is hierarchical clustering based on the distance of the feature amount vectors.

[0042] Here, Fig. 5 is an explanatory diagram of the cluster analysis process executed by the cluster analysis unit 12e. As shown as a dendrogram in Fig. 5, the cluster analysis unit 12e executes hierarchical clustering based on the distance of the feature vectors, so that image data having similar features are classified into the same cluster.

[0043] The number of clusters can be specified as desired. In the example of Figure 5, when the legs of the dendrogram are separated by separator line D1, the number of clusters is two (see clusters #1 and #2 in the figure). Similarly, when separated by separator line D2, the number of clusters is three. Similarly, when separated by separator line D3, the number of clusters is five.

[0044] Returning to the description of Fig. 2, the determination unit 12f identifies redundant data included in each cluster for each cluster based on the cluster information 11f that is the result of the cluster analysis performed by the cluster analysis unit 12e. The determination unit 12f also determines whether the identified redundant data satisfies a predetermined exclusion condition.

[0045] If the redundant data identified by the determination unit 12f satisfies a predetermined exclusion condition, the exclusion unit 12g excludes the redundant data from the image data set 11c, and generates a final data set 11g from which the redundant data has been excluded.

[0046] Here, Fig. 6 is an explanatory diagram (part 1) of the determination process executed by the determination unit 12f. Fig. 7 is an explanatory diagram (part 2) of the determination process executed by the determination unit 12f. Fig. 8 is an explanatory diagram (part 3) of the determination process executed by the determination unit 12f.

[0047] As shown in Fig. 6, the determination unit 12f identifies redundant data included in each cluster for each cluster based on the cluster information 11f, and assigns a redundant data flag to the identified redundant data. In the example of Fig. 6, the determination unit 12f sets the redundant data flag to "1" for the identified redundant data.

[0048] In identifying redundant data, as shown in FIG. 7, when the number of data in a cluster (N)=2, the determination unit 12f selects one of the image data p1 and p2 as redundant data.

[0049] 8, when N≧3, the determination unit 12f determines that the image data p1, p2, p3, ... are redundant data in the order of decreasing distance from the center of gravity G in the feature amount space of the cluster. When the maximum number of image data p to be left as learning data in the cluster (maximum number in the cluster) is T, the determination unit 12f specifies NT pieces of data that are close to the center of gravity G as redundant data. The value of T can be set arbitrarily.

[0050] In addition, by retaining image data p (p3 in the example of Figure 8) that is far from the center of gravity G, in other words, close to the cluster boundary, so that it is included in the final dataset 11g, it becomes possible to perform enhanced learning of features near the cluster boundary, which can contribute to improving the object detection performance of the object detection model.

[0051] Returning to the description of Fig. 2, the output unit 12h outputs the final data set 11g generated by the removal unit 12g to the learning device 20. The learning device 20 learns an object detection model by a machine learning algorithm using the final data set 11g output by the output unit 12h.

[0052] Next, a process procedure executed by the generating device 10 according to the embodiment will be described with reference to Fig. 9. Fig. 9 is a flowchart showing a process procedure executed by the generating device 10 according to the embodiment.

[0053] As shown in FIG. 9, the generating unit 12b generates an image data set 11c by cutting out a bounding box region B from each image P stored in the image DB 11a based on the generation parameter information 11b (step S101).

[0054] The learning unit 12c then learns the image classification model 11d using the image data set 11c (step S102). The feature acquisition unit 12d then infers the image data set 11c using the trained image classification model 11d (step S103) and acquires a feature vector by inference (step S104).

[0055] Then, the cluster analysis unit 12e executes hierarchical clustering based on the distances of the acquired feature vectors (step S105), and the determination unit 12f executes loop processing the number of times corresponding to the number of clusters (step S106).

[0056] In this loop process, the determination unit 12f acquires image data p in a cluster (step S1061) and determines whether or not there is redundant data (step S1062). If there is redundant data (step S1062, Yes), the determination unit 12f sets a redundant data flag for the identified redundant data (step S1063). If there is no redundant data (step S1062, No), the determination unit 12f does not set a redundant data flag.

[0057] After executing the loop process, the determining unit 12f determines whether or not the number of redundant data items for which the redundant data flag is set exceeds a predetermined threshold value (step S107). That is, it determines whether or not the above-mentioned predetermined exclusion condition is satisfied.

[0058] If the threshold is exceeded (step S107, Yes), the elimination unit 12g eliminates the redundant data from the image data set 11c (step S108) and ends the process. If the threshold is not exceeded (step S107, No), the elimination unit 12g does not eliminate the redundant data from the image data set 11c and ends the process.

[0059] By the way, using the generating device 10 described above, it is possible to configure a learning system 1 in a data collection system that collects various data from a vehicle.

[0060] 10 is a block diagram showing an example of the configuration of a learning system 1 according to an embodiment. The learning system 1 includes an in-vehicle device 100 and a data collection device 200.

[0061] The in-vehicle device 100 is a terminal device mounted on a vehicle for use. The vehicle is, for example, a vehicle having wheels, such as an automobile, a motorcycle, a train, or an automated guided vehicle. The vehicle is equipped with various sensors including a camera. The in-vehicle device 100 acquires various data from the various sensors. The various data may include, for example, the speed of the vehicle, the position of the vehicle, the time it takes for the vehicle to pass a specific position, and the like, in addition to images captured by the camera.

[0062] The in-vehicle device 100 may be used as a drive recorder or a navigation device, for example. However, the in-vehicle device 100 may be a device other than the drive recorder or the navigation device. The in-vehicle device 100 may be a device mounted on a vehicle, or may be a portable terminal that can be carried by a vehicle occupant. When the in-vehicle device 100 is a portable terminal, the in-vehicle device 100 may be, for example, a smartphone, a tablet, or the like.

[0063] The data collection device 200 is provided so as to be able to communicate with the in-vehicle device 100 via a network (not shown) such as the Internet or a mobile phone network. The data collection device 200 is configured, for example, as a cloud server that provides a cloud service via the network. The data collection device 200 accepts a data collection request from a data user (not shown) and provides the data user with data collected from the in-vehicle device 100 based on the accepted collection request. The data collection device 200 is usually provided so as to be able to communicate with a plurality of in-vehicle devices 100 and collects data from each of the in-vehicle devices 100.

[0064] As shown in Fig. 10, the in-vehicle device 100 includes a collection unit 101, a communication unit 102, and an in-vehicle model 106. The collection unit 101 collects various data obtained from various sensors mounted on the vehicle. The various data include captured images of the inside and outside of the vehicle captured by an in-vehicle camera. The collection unit 101 also transmits the collected various data to the data collection device 200 via the communication unit 102.

[0065] The communication unit 102 is realized by, for example, a network interface card (NIC) etc. The communication unit 102 is wirelessly connected to a network such as the Internet or a mobile phone network, and transmits and receives information to and from the data collection device 200 via the network.

[0066] The in-vehicle model 106 is a trained model that is mounted on a vehicle, and is, for example, an object detection model that detects objects included in an image captured by an in-vehicle camera.

[0067] The data collection device 200 includes a communication unit 201, a generation unit 202, a learning unit 203, and an in-vehicle model 204. The communication unit 201 is realized by, for example, a NIC, similar to the communication unit 102. The communication unit 201 is connected to a network, such as the Internet or a mobile phone network, by wire or wirelessly, and transmits and receives information to and from the in-vehicle device 100 via the network.

[0068] The generation unit 202 is a processing unit corresponding to the above-mentioned generation device 10. The generation unit 202 collects a group of images captured by the in-vehicle device 100, and generates a final data set 11g, which is a learning data set for object detection, based on the group of images.

[0069] The learning unit 203 is a processing unit corresponding to the above-mentioned learning device 20. The learning unit 203 learns an in-vehicle model 204, which is an object detection model to be mounted on a vehicle, by a machine learning algorithm using the final data set 11g generated by the generation unit 202.

[0070] When the learning is completed, the in-vehicle model 204 is transmitted to the in-vehicle device 100 via the communication unit 201. The in-vehicle device 100 receives the in-vehicle model 204 transmitted from the data collection device 200 via the communication unit 102, and stores it as the in-vehicle model 106. The collection unit 101 uses the in-vehicle model 106 to infer an object contained in an image captured by the in-vehicle camera.

[0071] Although not shown in FIG. 10, the various data transmitted from the vehicle-mounted device 100 to the data collection device 200 includes tag data and actual data.

[0072] As supplementary information including the above points, a series of steps until data is provided to a data user in a data collection system including learning system 1 will be specifically explained using Figs. 11 to 13. Fig. 11 is a diagram (part 1) showing an example of the operation of the data collection system. Fig. 12 is a diagram (part 2) showing an example of the operation of the data collection system. Fig. 13 is a diagram (part 3) showing an example of the operation of the data collection system.

[0073] As shown in FIG. 11, first, a data user specifies collection conditions using a user terminal 300 connected to the data collection device 200 .

[0074] At this time, the data collection device 200 also generates generation data to generate tag data T that is added to the actual data R to be collected and has characteristics as index data used for searching, understanding the overview, and selecting the actual data R. The generation data for the tag data T is generated based on the operation of the data user using a program and generation data stored in the user terminal 300 or the data collection device 200.

[0075] The specified collection conditions and the data for generating the generated tag data T are stored in the data collection device 200, and are also distributed to the vehicles that are the targets of data collection and stored in the in-vehicle device 100.

[0076] Next, each in-vehicle device 100 monitors output data of various sensors, and when an event occurs that satisfies the stored collection conditions, stores the actual data R in a storage device. Also, each in-vehicle device 100 generates tag data T corresponding to the actual data R based on the generation data of the stored tag data T and the actual data R, and stores the tag data T.

[0077] It is preferable that the tag data T is not simply a portion of the actual data R, but is meta-information to the extent that a data user can grasp the outline of the actual data R when referring to it and determine whether or not the actual data R is necessary.

[0078] Then, each in-vehicle device 100 uploads the tag data T to the data collection device 200 as needed, and the data collection device 200 stores the tag data T as needed. At this time, the actual data R is not uploaded to the data collection device 200. In addition, "as needed" as used herein does not only mean in real time (i.e., "instantly"), but also broadly includes a case where there is a certain degree of real-time nature, such as uploading all at once when the communication environment is ready. It may include not only when the communication environment is ready, but also when a certain amount of data has been accumulated, when a certain amount of time has passed since the previous upload, or when a certain distance has been traveled. It may also include the meaning of "periodically."

[0079] When a data user connects to the data collecting device 200 with the user terminal 300 to check the data collection status or collect the actual data R, meta-information based on the tag data T collected by the data collecting device 200 is displayed on the user terminal 300. Additionally, a UI (User Interface) screen for selecting the actual data R corresponding to each tag data T is displayed.

[0080] Then, as shown in FIG. 12, when a data user specifies tag data T corresponding to the actual data R that he or she wishes to collect using the user terminal 300, “instruction data” specifying the actual data R is transmitted to the corresponding vehicle-mounted device 100 via the data collection device 200.

[0081] 13, the specified real data R is selected and uploaded from each in-vehicle device 100 to the data collection device 200, and stored in the data collection device 200. Then, the data user accesses the real data R stored in the data collection device 200 using the user terminal 300 to view or download the real data R.

[0082] From the viewpoint of the storage capacity of the in-vehicle device 100, it is preferable that the actual data R uploaded to the data collection device 200 and the corresponding tag data T are deleted from the in-vehicle device 100 after the upload to the data collection device 200 is completed.

[0083] Here, an example of the tag data T will be described with reference to Fig. 14. Fig. 14 is a diagram showing an example of the tag data T. As shown in Fig. 14, the tag data T includes items such as "tag ID", "vehicle ID", "photographing date and time", and "inference result".

[0084] "Tag ID" is identification information for identifying tag data T. "Vehicle ID" is identification information for identifying a vehicle or an in-vehicle device 100. "Photography date and time" is the date and time when an image was captured by an in-vehicle camera. "Inference result" is the object detection result by the in-vehicle model 106, and is the label with the highest score.

[0085] 10 shows an example in which the data collection device 200 includes a generation unit 202 corresponding to the above-mentioned generation device 10 and a learning unit 203 corresponding to the learning device 20, and performs an integrated process from generation of a learning dataset to learning by the data collection device 200. However, the functions corresponding to the generation device 10 and the learning device 20 may be distributed between the in-vehicle device 100 and the data collection device 200, or may be integrated into the in-vehicle device 100.

[0086] These modified examples will be described with reference to Fig. 15 and Fig. 16. Fig. 15 is a block diagram showing a configuration example of a learning system 1A according to a first modified example. Fig. 16 is a block diagram showing a configuration example of a learning system 1B according to a second modified example. Note that Fig. 15 and Fig. 16 each correspond to Fig. 10, and therefore differences from Fig. 10 will be mainly described here.

[0087] 15, the in-vehicle device 100A of the learning system 1A according to the first modified example further includes a learning unit 105. The learning unit 105 is a processing unit equivalent to the above-mentioned learning device 20. On the other hand, the data collection device 200A does not include the above-mentioned learning unit 203.

[0088] In the learning system 1A according to the first modification, the generation unit 202 of the data collection device 200A collects a group of images captured by the in-vehicle device 100A, and generates a final data set 205, which is a learning data set for object detection, based on the group of images. The final data set 205 corresponds to the above-mentioned final data set 11g.

[0089] Then, the final data set 205 is transmitted to the in-vehicle device 100A via the communication unit 201. The in-vehicle device 100A receives the final data set 205 transmitted from the data collection device 200A via the communication unit 102, and stores it as the final data set 104.

[0090] Then, the learning unit 105 learns an in-vehicle model 106, which is an object detection model to be mounted on a vehicle, by a machine learning algorithm using the final data set 104. Then, the collection unit 101 infers an object included in an image captured by the in-vehicle camera, using the in-vehicle model 106 learned within the in-vehicle device 100A.

[0091] As a result, the functions corresponding to the generation device 10 and the learning device 20 can be distributed between the in-vehicle device 100A and the data collection device 200A to distribute the processing load, while preventing an increase in learning time and overlearning.

[0092] 16, the in-vehicle device 100B of the learning system 1B according to the second modified example further includes a generating unit 103 and a learning unit 105. The generating unit 103 is a processing unit corresponding to the above-mentioned generating device 10. On the other hand, the data collecting device 200B does not include the above-mentioned generating unit 202 and learning unit 203.

[0093] In the learning system 1B according to the second variant, the generation unit 103 of the in-vehicle device 100B collects a group of images taken by the in-vehicle device 100B itself, and generates a final dataset 104 based on the group of images, which serves as a learning dataset for object detection.

[0094] Then, the learning unit 105 learns an in-vehicle model 106, which is an object detection model to be mounted on a vehicle, by a machine learning algorithm using the final data set 104. Then, the collection unit 101 infers an object contained in an image captured by an in-vehicle camera, using the in-vehicle model 106, which has been subjected to processes from generation of a learning data set to learning within the in-vehicle device 100B.

[0095] As a result, the functions corresponding to the generating device 10 and the learning device 20 can be integrated into the in-vehicle device 100B, and the in-vehicle model 106 can be learned by self-learning of the in-vehicle device 100B. In this case, the in-vehicle model 106 can be a model specific to the in-vehicle device 100B.

[0096] As described above, the generating device 10 according to the embodiment (corresponding to an example of an "image dataset generating device") includes the generating unit 12b and the excluding unit 12g. The generating unit 12b generates an image dataset 11c by extracting bounding box regions B1-B5... (corresponding to an example of an "area") corresponding to an object detection range from the image set 3 and labeling each region with a class name. The excluding unit 12g excludes at least one of the image data p having similar features from the image dataset 11c.

[0097] Therefore, according to the generation device 10 according to the embodiment, it is possible to prevent an increase in learning time and overlearning.

[0098] The generating device 10 further includes a feature amount acquiring unit 12d. The feature amount acquiring unit 12d acquires feature amounts of each image data p by inputting the image data set 11c to an image classification model 11d (corresponding to an example of a "trained model") trained by machine learning using the image data set 11c. The exclusion unit 12g excludes image data p having similar features acquired by the feature amount acquiring unit 12d.

[0099] Therefore, according to the generating device 10 of the embodiment, it is possible to obtain features of each image data p by inference through regression analysis of the image dataset 11c using the image classification model 11d, and to characterize and eliminate redundant data based on such features.

[0100] The generating device 10 further includes a cluster analysis unit 12e that performs cluster analysis based on the feature acquired by the feature acquisition unit 12d. The exclusion unit 12g excludes at least one of the image data p that are classified into the same cluster by the cluster analysis unit 12e.

[0101] Therefore, the generating device 10 according to the embodiment enables cluster analysis based on similarity in feature space, regardless of the appearance of the image data p in a two-dimensional image space.

[0102] The feature amounts are feature amount vectors in the feature amount space of the image classification model 11d, and the cluster analysis unit 12e performs hierarchical clustering based on the distance of the feature amount vectors as the cluster analysis.

[0103] Therefore, according to the generating device 10 of the embodiment, the number of clusters can be set arbitrarily, and stable analysis results that are always consistent for each trial can be obtained. Also, cluster analysis based on similarity in feature space, regardless of the appearance of the image data p in a two-dimensional image space, becomes possible.

[0104] Furthermore, when three or more image data p exist in the same cluster, the exclusion unit 12g excludes them in order of the distance from the center of gravity G of the cluster.

[0105] Therefore, according to the generation device 10 according to the embodiment, it becomes possible to perform reinforced learning of features near the boundaries of clusters, which can contribute to improving the object detection performance of an object detection model.

[0106] Moreover, the data collection device 200 (corresponding to an example of a "learning device") according to the embodiment includes a generation unit 202 (corresponding to the "generation device 10") and a learning unit 203. The learning unit 203 learns an object detection model that detects an object by machine learning using a final data set 11g from which the image data p has been excluded by the exclusion unit 12g.

[0107] Therefore, according to the data collection device 200 according to the embodiment, the processes from generation of the learning data set to learning can be performed in an integrated manner. This also reduces the processing load on the in-vehicle device 100.

[0108] Moreover, the learning system 1B according to the second modified example is an in-vehicle system mounted on a vehicle, and includes a generation unit 103 (corresponding to the "generation device 10") and a learning unit 105. The learning unit 105 learns an object detection model that detects objects by machine learning using a final data set 11g from which image data p has been excluded by an exclusion unit 12g.

[0109] Therefore, according to the learning system 1B of the second modification, the functions corresponding to the generating device 10 and the learning device 20 are integrated into the in-vehicle device 100B, and the in-vehicle model 106 can be learned by the self-learning of the in-vehicle device 100B. In such a case, the in-vehicle model 106 can be a model specific to the in-vehicle device 100B.

[0110] In the above-described embodiment, the learning systems 1, 1A, and 1B included in the data collection system are given as an example of an in-vehicle system, but the learning systems 1, 1A, and 1B may be included in, for example, an automatic driving control system, etc. When included in an automatic driving control system, the in-vehicle devices 100, 100A, and 100B execute automatic driving control of the vehicle based on the detection results of the in-vehicle model 106.

[0111] In the above embodiment, deep learning is used as an example of the machine learning algorithm, but other algorithms may be used. For example, a Support Vector Machine (SVM) using a Histogram Of Gradient (HOG) feature may be used.

[0112] Further advantages and modifications may readily occur to those skilled in the art. Thus, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and equivalents thereof. [Explanation of symbols]

[0113] 1,1A,1B Learning System 10 Generator 11a Image DB 11b Generated parameter information 11c Image Dataset 11d Image Classification Model 11e Feature information 11f Cluster Information 11g Final Data Set 12a Collection Section 12b Generator 12c Learning Department 12d Feature acquisition unit 12e Cluster Analysis Section 12f Judgment section 12g exclusion part 12h Output section 20 Learning Device 100,100A,100B In-vehicle equipment 103 Generation part 104 Final Data Set 105 Learning Department 106 Vehicle Model 200, 200A, 200B Data Collection Device 202 Generation part 203 Learning Department 204 Vehicle Model 205 Final Data Set B, B1~B5 bounding box area G center of gravity P Image data p,p1~p3 Image data

Claims

1. An area corresponding to the object detection range is cut out from the image to generate learning image data, A learning image data set is generated that includes the plurality of learning image data generated; A plurality of pieces of training image data in the training image dataset are input into a trained model to obtain feature amounts of each piece of training image data; Executing a cluster analysis based on the feature amount of each of the learning image data; Calculating a distance from a center of gravity of a cluster to which each feature of the learning image data belongs based on a result of the cluster analysis; For the learning image data belonging to the same cluster, an exclusion priority is set in the order of distance from the center of gravity of the cluster; creating a learning image dataset by excluding the learning image data based on the exclusion priority; An image dataset generator comprising a controller.

2. A learning data is generated by labeling a class name on learning image data obtained by cutting out an area corresponding to an object detection range from an image, generating a learning data set consisting of a plurality of the generated learning data; Inputting a plurality of pieces of training image data in a training data set into a trained model to obtain feature amounts of each piece of training image data; Executing a cluster analysis based on the feature amount of each of the learning image data; Calculating a distance from a center of gravity of a cluster to which each feature of the learning image data belongs based on a result of the cluster analysis; For the learning image data belonging to the same cluster, an exclusion priority is set in the order of distance from the center of gravity of the cluster; creating a final training data set by excluding the training data based on the exclusion priority; Training an object detection model to detect objects by machine learning using the final training dataset; A learning device having a controller.

3. A learning system comprising an in-vehicle device and a data collection device, The in-vehicle device includes: Transmitting an image captured by an on-board camera to the data collection device; The data collection device includes: receiving the image transmitted from the in-vehicle device; generating learning data by labeling class names on learning image data obtained by cutting out an area corresponding to an object detection range from the image; generating a learning data set consisting of a plurality of the generated learning data; Inputting a plurality of pieces of training image data in a training data set into a trained model to obtain feature amounts of each piece of training image data; Executing a cluster analysis based on the feature amount of each of the learning image data; Calculating a distance from a center of gravity of a cluster to which each feature of the learning image data belongs based on a result of the cluster analysis; For the learning image data belonging to the same cluster, an exclusion priority is set in the order of distance from the center of gravity of the cluster; creating a final training data set by excluding the training data based on the exclusion priority; Training an object detection model to detect objects by machine learning using the final training dataset; Learning system.

4. An area corresponding to the object detection range is cut out from the image to generate learning image data, A learning image data set is generated that includes the plurality of learning image data generated; Inputting a plurality of pieces of training image data in a training image dataset into a trained model to obtain feature amounts of each piece of training image data; Executing a cluster analysis based on the feature amount of each of the learning image data; Calculating a distance from a center of gravity of a cluster to which each feature of the learning image data belongs based on a result of the cluster analysis; For the learning image data belonging to the same cluster, an exclusion priority is set in the order of distance from the center of gravity of the cluster; creating a learning image dataset by excluding the learning image data based on the exclusion priority; A method for generating an image dataset implemented by a controller.

Images