Feature data generation device and method, and machine learning device and method

Abstract

To provide a feature quantity data generation device and a method thereof, and a machine learning device and a method thereof that reduce learning time in machine learning.SOLUTION: In a data processor including a first learning data acquisition unit, a first coupling unit, a first learning unit, a second learning data acquisition unit, a second coupling unit, and a second learning unit, the second learning unit 60 acquires, as input data (IN_B), second coupled data composed of image data of a plurality of images each including a recognition target object. By compressing the input data IN_B with a learned encoder 32a, compressed data (E_B) including each feature quantity of a plurality of recognition target objects in a plurality of images are generated. The second learning unit 60 uses the compressed data E_B as input data to a neural network (NN) 61 to perform machine learning of the NN 61.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to a feature data generating device and method, and a machine learning device and method. [Background technology]

[0002] Mini-batch learning is often used when training an inference model for image recognition. In mini-batch learning, image data of multiple training images constituting training data is divided into mini-batches having a predetermined mini-batch size, and training is performed for each mini-batch. For example, when the number of pixels W in the horizontal direction and the number of pixels H in the vertical direction of a training image are both 100 and a color image in RGB format is used as a training image, the data size of one training image is (W×H×3), and a mini-batch is formed by combining image data of 32 training images in the mini-batch size direction. The mini-batch size in this case is (W×H×3×32).

[0003] For example, if the training data contains 10240 training images, then 320 mini-batch training runs will complete one training run for all training images, since 10240 / 32=320. In other words, the number of iterations is 320, and 320 mini-batch training runs correspond to one epoch. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-71808 A Summary of the Invention [Problem to be solved by the invention]

[0005] In the above method, if the number of training images included in one mini-batch is increased, the mini-batch size increases proportionally, but the number of times mini-batch learning is performed per epoch decreases. For example, if the mini-batch size is (W x H x 3 x 320), one training session for all training images is completed by performing mini-batch learning 32 times. In other words, one epoch is completed after 32 mini-batch learning sessions. By reducing the number of mini-batch learning sessions per epoch, the learning time of the inference model (for example, the time required for the value of the loss function to fall below a certain threshold) may be shortened.

[0006] However, since the memory capacity of a machine learning device is limited, the mini-batch size cannot be increased unconditionally. Although it depends on the size of each learning image, the number of learning images per mini-batch is often limited to about 32 in reality. Therefore, if the number of learning images per mini-batch increases beyond 32, the cost of the device will increase with the increase in the required memory capacity. It would be beneficial to be able to reduce the learning time without increasing the required memory capacity.

[0007] An object of the present invention is to provide a feature data generation device and method, and a machine learning device and method, that contribute to reducing learning time. [Means for solving the problem]

[0008] A feature data generating device according to the present invention has a configuration (first configuration) including an image data acquiring unit that acquires image data of a plurality of images, each of which includes a recognition target object, and a feature data generating unit that generates feature data including each of the features of a plurality of recognition target objects in the plurality of images by compressing the image data of the plurality of images.

[0009] In the feature data generating device according to the first configuration, the plurality of images may include two or more images captured by a predetermined camera in succession over time (second configuration).

[0010] The machine learning device of the present invention is configured (third configuration) to include a first combination unit that generates first combined data by combining image data of multiple first input images in the channel direction, a first learning unit that receives the first combined data and trains an autoencoder having an encoder that compresses the first combined data in the channel direction and a decoder that decompresses the compression, a second combination unit that generates second combined data by combining image data of multiple second input images in the channel direction, and a second learning unit that inputs the second combined data to the encoder after learning by the first learning unit, and inputs the compressed data output from the encoder to a neural network, thereby training the neural network.

[0011] In the machine learning device of the above third configuration, the second learning unit may be configured (fourth configuration) to train the neural network using teacher data including a plurality of label data corresponding to the plurality of second input images.

[0012] In the machine learning device according to the fourth configuration, the second learning unit may be configured to create an inference model capable of object detection by training the neural network (fifth configuration).

[0013] In the machine learning device according to the fifth configuration, each of the first input images and each of the second input images may include a recognition target object in the object detection (sixth configuration).

[0014] In the machine learning device according to any of the third to sixth configurations, the first combined data has image data of the plurality of first input images arranged in the channel direction, and the second combined data has image data of the plurality of second input images arranged in the channel direction, and in learning by the first learning unit, the encoder compresses the first combined data by reducing the number of dimensions of the first combined data in the channel direction, and in learning by the second learning unit, the encoder after learning by the first learning unit compresses the second combined data by reducing the number of dimensions of the second combined data in the channel direction, thereby obtaining the compressed data (seventh configuration).

[0015] In the machine learning device relating to the seventh configuration, the image data of each first input image and the image data of each second input image may include image data for multiple colors, and in the first combined data, the image data for the multiple colors of each first input image may be arranged in the channel direction, and in the second combined data, the image data for the multiple colors of each second input image may be arranged in the channel direction (eighth configuration).

[0016] A feature data generating method according to the present invention has a configuration (ninth configuration) including an image data acquiring step of acquiring image data of a plurality of images each including a recognition target object, and a feature data generating step of generating feature data including each feature of a plurality of recognition target objects in the plurality of images by compressing the image data of the plurality of images.

[0017] The machine learning method of the present invention has a configuration (tenth configuration) comprising: a first combining step of generating first combined data by combining image data of multiple first input images in a channel direction; a first learning step of training an autoencoder that receives the first combined data and has an encoder that compresses the first combined data in the channel direction and a decoder that decompresses the compression; a second combining step of generating second combined data by combining image data of multiple second input images in the channel direction; and a second learning step of inputting the second combined data to the encoder after learning by the first learning step, and inputting the compressed data output from the encoder into a neural network, thereby training the neural network (tenth configuration). Effect of the Invention

[0018] According to the present invention, it is possible to provide a data recording device and method that contribute to improving the convenience of data recording. [Brief description of the drawings]

[0019] [Figure 1] 1 is a configuration diagram of a data processing device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing the configuration of first learning data according to the embodiment of the present invention. [Diagram 3] FIG. 4 is a configuration diagram of second learning data according to the embodiment of the present invention. [Figure 4] FIG. 2 is a diagram showing one input image and corresponding label data according to the embodiment of the present invention. [Diagram 5] FIG. 2 is a diagram illustrating a configuration of one input image as a color image in RGB format according to the embodiment of the present invention. [Figure 6] FIG. 2 is a diagram showing a configuration of first combined data according to the embodiment of the present invention. [Figure 7] FIG. 2 is an explanatory diagram of the configuration and operation of an autoencoder according to an embodiment of the present invention. [Figure 8] FIG. 11 is a diagram showing a configuration of second combined data according to the embodiment of the present invention. [Figure 9] FIG. 4 is an explanatory diagram of a learning operation of a second learning unit according to the embodiment of the present invention. [Figure 10] FIG. 4 is an explanatory diagram of input data to the neural network of the second learning unit in the embodiment of the present invention. [Figure 11] FIG. 2 is a diagram for explaining the contents of training data according to an embodiment of the present invention. [Figure 12] 4 is a flowchart illustrating an operation of the data processing device according to the embodiment of the present invention. [Figure 13] FIG. 11 is a diagram for explaining the effect of data compression in the embodiment of the present invention. [Figure 14] 1 is a configuration diagram of a feature data generating device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, examples of the embodiments of the present invention will be described in detail with reference to the drawings. In each of the drawings, the same parts are given the same reference numerals, and duplicated descriptions of the same parts are omitted as a rule. In this specification, for the sake of simplifying the description, a symbol or a code referring to information, a signal, a physical quantity, a member, etc. may be written, and the name of the information, signal, physical quantity, member, etc. corresponding to the symbol or code may be omitted or abbreviated. For example, the second learning data acquisition unit referred to by "40" described later (see FIG. 1) may be written as the second learning data acquisition unit 40 or may be abbreviated as the acquisition unit 40, but they all refer to the same thing.

[0021] Although details will be described later, in this embodiment, a trained encoder 32a capable of extracting features included in an image is generated using the first training data (see FIG. 9). Next, the trained encoder 32a is used to generate data (compressed data) in which features of a recognition target object are extracted from the second training data. When the trained encoder 32a extracts the features of a recognition target object from the second training data, a technique called compression is used. Next, the NN61 is trained using the data (compressed data) in which the features of the recognition target object are extracted. The NN61 becomes an inference model for object detection through training. The data (compressed data) in which the features of a recognition target object are extracted is used for training the NN61, which can contribute to reducing the training time of the NN61. This will be described in detail below.

[0022] FIG. 1 shows a configuration diagram of a data processing device 1 according to this embodiment. The data processing device 1 is an example of a machine learning device. The data processing device 1 includes a first learning data acquisition unit 10, a first combination unit 20, a first learning unit 30, a second learning data acquisition unit 40, a second combination unit 50, and a second learning unit 60. The data processing device 1 may be configured as a single computer device, or may be configured as multiple physically separated computer devices. The data processing device 1 may be configured by utilizing so-called cloud computing.

[0023] The first learning data acquisition unit 10 acquires first learning data including image data of a plurality of images. Since the image data of each image constituting the first learning data is input to the first combination unit 20, each image constituting the first learning data is referred to as a first input image. As shown in FIG. 2, the first learning data includes image data of a total of P first input images IA[1] to IA[P]. P is an arbitrary integer equal to or greater than 2, and has a value of, for example, several tens to several thousands. Note that the first input image IA[i] may be simply written as an input image IA[i]. i represents an arbitrary integer.

[0024] The second learning data acquisition unit 40 acquires second learning data including image data of a plurality of images. Since the image data of each image constituting the second learning data is input to the second combining unit 50, each image constituting the second learning data is referred to as a second input image. As shown in FIG. 3, the second learning data includes image data of a total of Q second input images IB[1] to IB[Q]. Q is an arbitrary integer of 2 or more, and for example, has a value of several thousand to several tens of thousands. Note that the second input image IB[i] may be simply denoted as the input image IB[i]. In the present embodiment, the number of images included in the second learning data is larger than the number of images included in the first learning data. That is, “P<Q” holds.

[0025] Note that any image such as the first input image or the second input image includes the image data of the image and other data (hereinafter referred to as additional data). An arbitrary image may be an image captured by a camera, and the additional data for a certain image includes data that is not image data among the image, and further includes shooting time information indicating the shooting time of the image.

[0026] As will be described later, in the data processing device 1, an inference model (algorithm) is created through the learning of the second learning unit 60, and the inference model can perform object detection as image recognition. In object detection, position identification for specifying the position of an object in an image to be recognized and class identification for specifying the class (type) of the object in the image to be recognized are performed. Each first input image and each second input image include one or more objects to be recognized. In the present embodiment, an object refers to a recognition target object that is the target of image recognition in object detection. Some of the first input images may not include a recognition target object. Similarly, some of the second input images may not include a recognition target object. In addition, one or more of the first input images may include objects other than the recognition target object. Similarly, one or more of the second input images may include objects other than the recognition target object.

[0027] In this embodiment, the inclusion of image data of an object in an image may be expressed as the object being included or present in the image. Similarly, the inclusion of image data of an object in a focused image region (e.g., an object region described below) in an image may be expressed as the object being included or present in the focused image region.

[0028] The second training data includes label data for each second input image. In the second training data, the label data associated with the second input image IB[i] is referred to by the symbol "LB[i]". The label data LB[i] includes, for each object included in the second input image IB[i], position information that identifies the position of the object and class information that identifies the class of the object.

[0029] FIG. 4 shows an input image 610. The input image 610 is an example of the second input image IB[i]. The input image 610 in FIG. 4 includes three objects 611 to 613. The objects 611, 612, and 613 are a vehicle, a person, and a traffic light, respectively, and all of them are objects to be recognized. Here, the vehicles, people, and traffic lights are classified into first, second, and third classes, and the inference model is capable of performing object detection for objects of multiple classes including the first to third classes. Note that, here, the vehicle is assumed to be an automobile capable of running on a road.

[0030] 4, an object region 611B surrounding the image of object 611, an object region 612B surrounding the image of object 612, and an object region 613B surrounding the image of object 613 are set. The object region of a certain object is a rectangular region (preferably the smallest rectangular region) surrounding the image of the object, and is also called a bounding box.

[0031] The label information 620 corresponding to the input image 610 in FIG. 611 and class information CLS 611 and position information POS 612 and class information CLS 612 and position information POS 613and class information CLS 613 If the input image 610 is the second input image IB[i], the label information 620 is the label information LB[i]. 611 , POS 612 , POS 613 respectively represent the positions of object region 611B, object region 612B, and object region 613B in input image 610. In detail, the coordinate values ​​(corresponding to coordinate values ​​(x1, y1) and (x2, y2) in FIG. 4) of two end points of one diagonal line in a rectangular region as object region 611B are defined as position information POS 611 The same applies to other location information. Class information CLS 611 , CLS 612 , CLS 613 respectively represent the class to which the object 611 belongs, the class to which the object 612 belongs, and the class to which the object 613 belongs. In the example of FIG. 4, the class information CLS 611 , CLS 612 , CLS 613 represent the first class to which vehicles belong, the second class to which humans belong, and the third class to which traffic lights belong, respectively.

[0032] For example, the first input images IA[1] to IA[P] and the second input images IB[1] to IB[Q] may be selected from images captured by a camera mounted on a vehicle such as an automobile. The first input images IA[1] to IA[P] and the second input images IB[1] to IB[Q] may partially overlap.

[0033] The first learning data acquisition unit 10 may be a functional block that creates the first learning data by itself, or may receive input of pre-created first learning data via wired or wireless communication from an external device (not shown) different from the data processing device 1. Similarly, the second learning data acquisition unit 40 may be a functional block that creates the second learning data by itself, or may receive input of pre-created second learning data from an external device (not shown) different from the data processing device 1 via wired or wireless communication.

[0034] Each of the first input images and each of the second input images is a two-dimensional still image having a size in the horizontal and vertical directions. One or more of the first input images may be frames of a moving image. Similarly, one or more of the second input images may be frames of a moving image. The number of pixels in the horizontal direction in each of the first input images and each of the second input images is represented by W, and the number of pixels in the vertical direction in each of the first input images and each of the second input images is represented by H. Then, each of the first input images and each of the second input images consists of (W×H) pixels. Also, it is assumed that the first input image and each of the second input images are color images expressed in RGB format. That is, each pixel of the first input image and each of the second input images has an R signal representing a red signal component, a G signal representing a green signal component, and a B signal representing a blue signal component.

[0035] Then, an input image 650, which is any one of the first input images or any one of the second input images, can be considered to be composed of a red grayscale image 650R consisting of (W×H) pixels and having only an R signal as a color signal, a green grayscale image 650G consisting of (W×H) pixels and having only a G signal as a color signal, and a blue grayscale image 650B consisting of (W×H) pixels and having only a B signal as a color signal, as shown in Fig. 5. The images 650R, 650G, and 650B are arranged in a channel direction different from both the horizontal and vertical directions. The number of types of color signals constituting the image data of the input image 650 is represented by "C". Here, "C=3".

[0036] Referring again to FIG. 1, the first combining unit 20 divides the first input images IA[1] to IA[P] into mini-batches having a predetermined mini-batch size. Then, for each mini-batch, the first combining unit 20 generates first combined data by combining multiple first input images belonging to the mini-batch in the channel direction. Here, the mini-batch size is assumed to be the data size of N first input images. N is an arbitrary integer of 2 or more, for example, "N=32". (P / N) pieces of first combined data are formed from the first input images IA[1] to IA[P]. (P / N) is an arbitrary integer of 2 or more.

[0037] FIG. 6 shows one first combined data (i.e., one mini-batch structure) generated by the first combining unit 20. The first combined data shown in FIG. 6 is composed of image data of input images IA[i] to IA[i+N], and the image data of input images IA[i] to IA[i+N] are arranged in the channel direction. Each of the input images IA[i] to IA[i+N] is composed of a red grayscale image, a green grayscale image, and a blue grayscale image arranged in the channel direction. Therefore, in the first combined data, image data for multiple colors of the input image IA[i] (image data of the red grayscale image, the green grayscale image, and the blue grayscale image), image data for multiple colors of the input image IA[i+1] (image data of the red grayscale image, the green grayscale image, and the blue grayscale image), ..., and image data for multiple colors of the input image IA[i+N] (image data of the red grayscale image, the green grayscale image, and the blue grayscale image) are arranged in the channel direction.

[0038] Therefore, the first combined data is equivalent to (C×N) monochrome two-dimensional images, each consisting of (H×W) monochrome pixels, arranged along the channel direction. The first combined data has a data volume of (W×H×C×N) monochrome pixels. In the first combined data, the number of channels is (C×N), and therefore the number of dimensions in the channel direction is (C×N).

[0039] Referring again to FIG. 1, the first learning unit 30 has a neural network 31 (hereinafter referred to as NN31), and performs machine learning of the NN31 using the first learning data. At this time, machine learning of the NN31 is performed in units of mini-batches. That is, (P / N) pieces of first combined data based on the first learning data are sequentially used as input data to the NN31, and machine learning of the NN31 is performed in units of mini-batches (mini-batch learning is performed). The machine learning in the first learning unit 30 may be classified as deep learning, and therefore the NN31 may be a deep neural network. The machine learning in the first learning unit 30 is unsupervised learning, and an autoencoder is formed by the NN31. That is, the first learning unit 30 trains an autoencoder (in other words, an autoencoder is created by training the NN31).

[0040] FIG. 7 shows the configuration of an autoencoder. The NN 31 forming the autoencoder includes an encoder 32 and a decoder 33. The type of the autoencoder here is arbitrary, and may be, for example, a variational autoencoder (VAE) or a convolutional autoencoder (CAE). The first combined data is input to the encoder 32 as input data IN_A, and the encoder 32 generates compressed data E_A by compressing the input data IN_A. The decoder 33 obtains output data OUT_A by restoring the compressed data E_A (i.e., by restoring the compression by the encoder 32). In the machine learning in the first learning unit 30, each parameter (bias and weight) of the NN 31 is adjusted so that the output data OUT_A matches the input data IN_A.

[0041] In this case, the encoder 32 is designed so that the input data IN_A (thus the first combined data) is compressed in the channel direction, and the decoder 33 is designed so that the compressed data E_A is restored in the channel direction. In other words, the compression by the encoder 32 corresponds to a dimension reduction in the channel direction, and the encoder 32 reduces the number of dimensions in the channel direction of the input data IN_A (thus the first combined data) from "(C×N)" to "J". In other words, the number of dimensions in the channel direction of the input data IN_A (thus the first combined data) is "(C×N)", and the encoder 32 performs a dimension reduction of the input data IN_A in the channel direction to obtain compressed data E_A with a number of dimensions in the channel direction of "J". It can also be considered that the number of channels is reduced from "(C×N)" to "J" in the encoder 32.

[0042] "C×N>J". For example, if "(C, N, J)=(3, 32, 3)", the number of dimensions in the channel direction of the input data IN_A (hence the first combined data) is reduced from 96 dimensions to 3 dimensions in the encoder 32. In this case, since "3 / (3×32)=1 / 32", the data size of the compressed data E_A becomes 1 / 32 of the data size of the input data IN_A.

[0043] The first learning unit 30 performs machine learning of the NN 31 until the training error (value of the loss function) of the NN 31 functioning as an autoencoder becomes equal to or less than a predetermined value. Hereinafter, the encoder 32 after the completion of this machine learning is specifically referred to as a trained encoder 32a (see FIG. 9).

[0044] Referring again to FIG. 1, the second combining unit 50 divides the second input images IB[1] to IB[Q] into mini-batches having a predetermined mini-batch size. The mini-batch size in the second combining unit 50 is the same as the mini-batch size in the first combining unit 20. Therefore, the mini-batch size in the second combining unit 50 is the data size of N sheets of second input images (for example, "N=32"). Then, the second combining unit 50 generates second combined data for each mini-batch by combining multiple second input images belonging to the mini-batch in the channel direction. (Q / N) pieces of second combined data are formed from the second input images IB[1] to IB[Q]. (Q / N) is an arbitrary integer of 2 or more, and has a value of, for example, several hundred to several thousand.

[0045] FIG. 8 shows one second combined data (i.e., the structure of one mini-batch) generated by the second combining unit 50. The structure of the second combined data is similar to that of the first combined data. That is, the second combined data shown in FIG. 8 is composed of image data of the input images IB[i] to IB[i+N], and the image data of the input images IB[i] to IB[i+N] are arranged in the channel direction. Each of the input images IB[i] to IB[i+N] is composed of a red shading image, a green shading image, and a blue shading image arranged in the channel direction. Therefore, in the second combined data, image data for multiple colors of the input image IB[i] (image data of the red shading image, the green shading image, and the blue shading image), image data for multiple colors of the input image IB[i+1] (image data of the red shading image, the green shading image, and the blue shading image), ..., and image data for multiple colors of the input image IB[i+N] (image data of the red shading image, the green shading image, and the blue shading image) are arranged in the channel direction.

[0046] Therefore, the second combined data is equivalent to (C×N) monochrome two-dimensional images, each consisting of (H×W) monochrome pixels, arranged along the channel direction. The second combined data has a data volume of (W×H×C×N) monochrome pixels. In the second combined data, the number of channels is (C×N), and therefore the number of dimensions in the channel direction is (C×N).

[0047] Referring again to FIG. 1, the second learning unit 60 has a neural network 61 (hereinafter referred to as NN61), and performs machine learning of the NN61 using the second learning data. The machine learning in the second learning unit 60 may be classified as deep learning, and therefore the NN61 may be a deep neural network. The machine learning in the second learning unit 60 is supervised learning, and an inference model for object detection is formed by the NN61. That is, the second learning unit 60 trains an inference model for object detection (in other words, an inference model capable of object detection is created by training the NN61).

[0048] Machine learning by the second learning unit 60 will be described with reference to FIG. 9. The above-mentioned trained encoder 32a is used for machine learning by the second learning unit 60. The second combined data is input to the trained encoder 32a as input data IN_B, and the trained encoder 32a generates compressed data E_B by compressing the input data IN_B. In compression by the trained encoder 32a, the number of dimensions in the channel direction of the input data IN_B (hence the second combined data) is reduced from "(C×N)" to "J". In other words, the number of dimensions in the channel direction of the input data IN_B (hence the second combined data) is "(C×N)", and the trained encoder 32a reduces the dimensions of the input data IN_B in the channel direction to obtain compressed data E_B with a number of dimensions in the channel direction of "J". It can also be considered that the number of channels is reduced from "(C×N)" to "J" by the trained encoder 32a.

[0049] A plurality of pieces of second combined data based on the second input images IB[1] to IB[Q] are sequentially input to the trained encoder 32a as input data IN_B, whereby a plurality of pieces of compressed data E_B based on the plurality of second combined data are obtained.

[0050] In the second learning unit 60, machine learning of the NN61 is performed using the compressed data E_B as input data to the NN61. At this time, machine learning of the NN61 is performed in units of mini-batches (i.e., mini-batch learning is performed). The mini-batch size in the machine learning of the NN61 and the mini-batch size in the machine learning of the NN31 may be different, but here they are assumed to be the same. In that case, the mini-batch size in the machine learning of the NN61 is the data size of N sheets of the second input image, and the data size of N sheets of the second input image is (W×H×C×N).

[0051] Fig. 10 shows an overview of the flow up to generation of input data to the NN 61. In Fig. 10, data DTa includes "(C x N) / J" sets of image data of N second input images. Data DTb consists of "(C x N) / J" pieces of input data IN_B, i.e., "(C x N) / J" pieces of second combined data. Data DTc consists of "(C x N) / J" pieces of compressed data E_B based on data DTb.

[0052] Data DTb is obtained by inputting data DTa to the second combining unit 50. That is, by sequentially inputting "(C×N) / J" sets of image data of N second input images to the second combining unit 50, "(C×N) / J" pieces of second combined data are output from the second combining unit 50. The data size of each piece of second combined data is (W×H×C×N). Therefore, the data size of data DTb is "(W×H×C×N)×(C×N) / J". The data size of data DTa is also the same.

[0053] Each second combined data is input to the trained encoder 32a as input data IN_B to generate compressed data E_B for each second combined data, resulting in data DTc consisting of "(C×N) / J" pieces of compressed data E_B. Since the number of dimensions in the channel direction is reduced from "(C×N)" to "J" in the trained encoder 32a, the data size of one piece of compressed data E_B is (W×H×J). Therefore, the data size of data DTc is (W×H×C×N).

[0054] Compressed data E_B with a data size of (W×H×C×N) is input to NN61 as data for one mini-batch learning. This is equivalent to inputting information on “N×(C×N) / J” input images to NN61 in one mini-batch learning of NN61. For example, in a numerical example where “(C,N,J)=(3,32,3)” and “Q=10240” are satisfied, 32 2 The information of the input image is input to the NN61. 2 = 10”, one round of learning using all of the second input images that make up the second learning data is completed by performing mini-batch learning of NN61 10 times (i.e., the number of iterations is 10).

[0055] In a hypothetical case in which the image data of N second input images themselves are input to the NN61, in order to complete one learning session using all the second input images that make up the second learning data, mini-batch learning of the NN61 needs to be performed 320 times in the above numerical example, which results in a longer learning time compared to the data processing device 1.

[0056] In machine learning of the NN61 in the second learning unit 60, the NN61 generates output data OUT_B based on compressed data E_B having a mini-batch size (see FIG. 9). The second learning unit 60 derives a value of a loss function corresponding to the error between the output data OUT_B and the teacher data for each mini-batch (for each mini-batch learning), and adjusts the parameters (weights and biases) of the NN61 using the backpropagation method so that the value of the loss function is reduced. Machine learning of the NN61 (i.e., machine learning of an inference model for object detection) is performed until the value of the loss function becomes equal to or less than a predetermined threshold.

[0057] In the mini-batch learning of NN61, the teacher data is composed of label data for all the second input images used in the mini-batch learning. For example, in a certain mini-batch learning, if the data DTa (see FIG. 10) is composed of image data of the second input images IB[1] to IB

[1024] , the teacher data in the mini-batch learning is composed of label data LB[1] to LB

[1024] . That is, for example, if the data DTa contains image data of the second input images IB[1] and IB[2], as shown in FIG. 11, the sum of the corresponding label data LB[1] and LB[2] is included in the teacher data corresponding to the data DTa (only the information of the label data LB[1] and LB[2] is shown in FIG. 11).

[0058] The NN61, which should function as an inference model, performs position identification and class identification of objects in each input image constituting the data DTa during machine learning using the data DTa, and outputs the results of position identification and class identification as output data OUT_B. The value of the loss function is derived by comparing this output data OUT_B with the teacher data corresponding to the data DTa.

[0059] 12 shows an operation flowchart of the data processing device 1. First, in step S1, the first learning data acquisition unit 10 acquires the first learning data. Next, in step S2, the first combination unit 20 generates first combined data based on the first learning data. Next, in step S3, the first learning unit 30 trains an autoencoder based on the first combined data (i.e., trains the NN 31), thereby creating a trained encoder 32a. Next, in step S4, the second learning data acquisition unit 40 acquires the second learning data. Note that the timing of acquiring the second learning data is arbitrary as long as it is before step S5.

[0060] After acquiring the second learning data, in step S5, the second combination unit 50 generates second combined data based on the second learning data. At this time, the above-mentioned teacher data is also created. The teacher data may be created by the second combination unit 50 or the second learning unit 60. Thereafter, in step S6, the second combined data is input to the trained encoder 32a to generate compressed data (E_B), and the second learning unit 60 trains an inference model for object detection based on the generated compressed data (E_B) (in other words, an inference model for object detection is created by training the NN 61).

[0061] In this embodiment, the amount of information of data per mini-batch in the second learning unit 60 can be increased by compressing the learning data (second learning data) as described above. That is, compared to a hypothetical case in which the image data of N second input images themselves are input to the NN 61, the amount of information of data per mini-batch in the second learning unit 60 increases by "(C×N) / J" times (for example, increases by 32 times). Therefore, compared to the hypothetical case, it is possible to shorten the learning time in the second learning unit 60 (for example, the time required for the value of the loss function of the inference model by the NN 61 to become equal to or less than a predetermined threshold). From another perspective, if the learning time is considered to be constant, the required memory capacity can also be reduced.

[0062] FIG. 13 shows the data size compression effect and the like in the case of "(C×N) / J=3×32 / 3=32". In the numerical example of FIG. 13, the data size (data amount) of the input data IN_B is compressed to 1 / 32 by using the trained encoder 32a, and compressed data E_B is obtained. Therefore, if the learning time per unit data amount of data input to the NN61 is considered to be constant, the learning time of the NN61 is shortened to 1 / 32 compared to the virtual case. In addition, the number of iterations in the learning of the NN61 is also reduced to 1 / 32 of the number of iterations required in the virtual case. On the other hand, when the learning time of the NN61 according to this embodiment is made the same as the learning time related to the virtual case, it is possible to reduce the required memory capacity to 1 / 32 of the virtual case.

[0063] Since each first input image includes a recognition target object of the inference model, in the autoencoder, the feature amount of the recognition target object of each first input image is extracted from the first combined data (IN_A) and included in the compressed data E_A. In other words, the autoencoder is trained so that the feature amount of the recognition target object is extracted from the input image including the recognition target object, and the trained encoder 32a is configured. Therefore, if the second combined data (IN_B) based on the second input image including the recognition target object is input to the trained encoder 32a, the feature amount of the recognition target object in each second input image is extracted by the trained encoder 32a and included in the compressed data E_B. By inputting this compressed data E_B to the inference model (NN61), efficient learning that contributes to shortening the learning time is possible.

[0064] When considering the function of the data processing device 1 from the viewpoint of the feature of the object to be recognized, the data processing device 1 can be considered to function as or include the feature data generating device 2 of FIG. 14. The feature data generating device 2 includes an image data acquiring unit 2A that acquires a plurality of images II each including the object to be recognized, and a feature data generating unit 2B that generates feature data including each feature of the plurality of objects to be recognized in the plurality of images II by compressing the image data of the plurality of images II. The plurality of images II related to the acquiring units 2A and 2B correspond to a plurality of second input images. The acquiring unit 2A corresponds to the acquiring unit 40 of FIG. 1, and the generating unit 2B corresponds to a functional block including the combining unit 50 of FIG. 1 and the trained encoder 32a of FIG. 9. The feature data generated by the generating unit 2B corresponds to the compressed data E_B.

[0065] That is, the feature data generating device 2 acquires a plurality of second input images each including a recognition target object, and generates feature data (EN_B) including each feature of a plurality of recognition target objects in the plurality of second input images. If the inference model (NN61) for object detection is trained using this feature data, efficient training that contributes to shortening the training time becomes possible.

[0066] Here, the above-mentioned multiple images II may include two or more images taken continuously in time by a specified camera (not shown). That is, at least a part of the second input images IB[1] to IB[Q] may be two or more images taken continuously in time by a specified camera. The specified camera takes an image of the state (subject) within its own shooting area, and generates image data of the camera image, which is the captured image. At this time, the specified camera periodically shoots at a specified frame rate. Then, the specified camera acquires multiple camera images arranged in time series at intervals of the reciprocal of the frame rate. The multiple camera images arranged in time series (hereinafter referred to as a camera image sequence) correspond to two or more images taken continuously in time by the specified camera.

[0067] The predetermined camera may be a fixed camera fixed at a certain location. In this case, it is expected that the scenery in the camera images (parts other than the object to be recognized) will hardly change in the camera image sequence, and only vehicles and people as objects to be recognized will move in the camera image sequence. In this case, the compression effect by the encoder 32 (trained encoder 32a) will be enhanced, making it possible to efficiently extract the feature amount of the object to be recognized, and thus promoting efficient learning of the inference model (NN61). The predetermined camera may be a camera mounted on a moving body such as a vehicle.

[0068] The image data acquisition unit 2A may extract two or more images taken consecutively in time from a collection of images taken by a specified camera based on the shooting time information included in the additional data of each image taken by the specified camera. For example, when a first captured image and a second captured image are included in the images taken by the specified camera, and the time difference between the shooting time of the first captured image and the shooting time of the second captured image is equal to or less than a specified time, the first and second captured images may be extracted as two images taken consecutively in time.

[0069] Supplementary matters, applied techniques, modified techniques, etc. to the above-mentioned configuration are given below.

[0070] Each of the first input images and each of the second input images may be a monochrome image (a grayscale image that does not have color information). In this case, "C=1".

[0071] The encoder 32 (including the trained encoder 32a) compresses the input data (IN_A or IN_B) in the channel direction, but at this time, the input data may be compressed in the horizontal or vertical direction of the input image.

[0072] The data processing device 1 includes, as hardware, a central processing unit (CPU), which is an arithmetic processing device, a graphics processing unit (GPU), a read only memory (ROM), and a random access memory (RAM), etc. The data processing device 1 may realize the functions of each part shown in Fig. 1 by executing a program stored in the ROM or a program acquired from another device through communication with the CPU, and thus may realize each process of steps S1 to S6 in Fig. 12.

[0073] The inference model created by the data processing device 1 may be applied to an in-vehicle device (not shown). The inference model created by the data processing device 1 is a type of electronic device mounted on a vehicle such as an automobile. In this case, the inference model formed by the NN61 after machine learning of the NN61 by the second learning unit 60 may be applied to the inference model created by the inference model. Then, the inference result may be used for automatic driving or driving assistance that may be implemented in the vehicle.

[0074] The data processing device 1 itself may be an in-vehicle device. Depending on the vehicle (e.g., a broadcasting vehicle), an in-vehicle device having abundant computing resources may be installed, and in this case, the data processing device 1 itself may be an in-vehicle device.

[0075] In addition, a part or all of the processing executed by the data processing device 1 may be realized by a mixture of software and hardware. A computer program for causing a computer to execute the above-mentioned method and a computer-readable recording medium having the program recorded thereon are included in the scope of this embodiment. Here, the computer-readable recording medium is, for example, a flexible disk, a hard disk, a CD-ROM, an MO, a DVD, a DVD-ROM, a DVD-RAM, a large-capacity DVD, a next-generation DVD, or a semiconductor memory.

[0076] The embodiments of the present invention can be modified in various ways as appropriate within the scope of the technical ideas set forth in the claims. The above embodiments are merely examples of the present invention, and the meanings of the terms of the present invention and each component are not limited to those described in the above embodiments. The specific numerical values ​​shown in the above description are merely examples, and can be changed to various numerical values ​​as a matter of course. [Explanation of symbols]

[0077] 1 Data Processing Device 2. Feature data generator 2A Image data acquisition section 2B Feature data generation section 10 First learning data acquisition unit 20 1st joint 30 First Learning Section 31 Neural Networks (Autoencoders) 32 Encoder 33 Decoder 40 Second learning data acquisition unit 50 Second joint 60 Second Learning Section 61 Neural Networks (Inference Models)

Claims

1. A machine learning device including a calculation processing unit that acquires image data of a plurality of images each including a recognition target object, and generates feature amount data including feature amounts of each of the plurality of recognition target objects in the plurality of images by compressing the image data of the plurality of images, the plurality of images including two or more images captured consecutively in time by a predetermined camera, The arithmetic processing unit is a first combining process for generating first combined data by combining image data of a plurality of first input images each including a recognition target object in a channel direction; a first learning process for learning an autoencoder using the first combined data, the autoencoder including an encoder for compressing the first combined data in the channel direction and a decoder for decompressing the compressed data; a second combining process for generating second combined data by combining image data of a plurality of second input images, which are the plurality of images, in the channel direction; a second learning process for inputting the second combined data to the encoder after learning by the first learning process, thereby generating compressed data output from the encoder as the feature amount data, and training a neural network using the compressed data. , machine learning device.

2. In the second learning process, the arithmetic processing unit trains the neural network using teacher data including a plurality of label data corresponding to the plurality of second input images. The machine learning device according to claim 1 .

3. In the second learning process, the arithmetic processing unit creates an inference model capable of object detection by training the neural network. The machine learning device according to claim 2 .

4. In the first combined data, image data of the plurality of first input images are arranged in the channel direction; In the second combined data, image data of the plurality of second input images are arranged in the channel direction; In the learning in the first learning process, the encoder compresses the first combined data by reducing a number of dimensions in a channel direction of the first combined data; In the learning in the second learning process, the encoder after learning in the first learning process reduces the number of dimensions in the channel direction of the second combined data, thereby compressing the second combined data, thereby obtaining the compressed data. The machine learning device according to any one of claims 1 to 3.

5. The image data of each of the first input images and the image data of each of the second input images include image data for a plurality of colors, In the first combined data, image data for the plurality of colors of each first input image is arranged in the channel direction, In the second combined data, the image data for the plurality of colors of each second input image is arranged in the channel direction. The machine learning device according to claim 4 .

6. A machine learning method for learning a neural network through a process of acquiring image data of a plurality of images each including a recognition target object, and generating feature amount data including feature amounts of each of the plurality of recognition target objects in the plurality of images by compressing the image data of the plurality of images, the plurality of images including two or more images captured consecutively in time by a predetermined camera, the machine learning method comprising: a first combining step of generating first combined data by combining image data of a plurality of first input images each including a recognition target object in a channel direction; a first learning step of learning an autoencoder using the first combined data, the autoencoder having an encoder that compresses the first combined data in the channel direction and a decoder that restores the compression; a second combining step of generating second combined data by combining image data of a plurality of second input images, which are the plurality of images, in the channel direction; a second learning step of inputting the second combined data to the encoder after learning in the first learning step to generate compressed data output from the encoder as the feature amount data, and learning the neural network using the compressed data. ,machine learning methods.

Images