Machine learning apparatus and machine learning method

Abstract

To properly carry out learning with a convolutional neural network (CNN) while avoiding loss or disappearance of features of a target object.SOLUTION: A machine learning apparatus is configured to extract features of target data using CNN, to perform learning on the target data, the CNN including: a first convolutional layer configured to differentiate features of the target data; a second convolutional layer configured to increase the amount of information from the differentiated target data; a max pooling layer configured to compress the target data with increased amount of information, other than the maximum value; an upsampling layer configured to increase the size of the compressed target data for upsampling; a third convolutional layer configured to convert the upsampled target data to target data of a predetermined number of channels; and a fourth convolutional layer configured to directly convert the target data prior to the first convolutional layer to the target data of the predetermined number of channels and add the target data after the third convolutional layer and the target data after the fourth convolutional layer.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a neural network that extracts features by performing calculations (learning, estimation) based on various collected data, and in particular to a machine learning device and a machine learning method that use a convolutional neural network. [Background technology]

[0002] In recent years, advanced technologies such as artificial intelligence (AI) and information and communications technology (ICT) have been utilized to put into practical use technologies that efficiently perform processes such as learning, estimation, recognition, and judgment. One of these is machine learning, a learning method performed by AI. In machine learning, a machine (computer) learns on its own using a large amount of data provided, and based on the learning results (trained model), optimizes output data for input data and performs estimation and prediction based on the output data. One example of such machine learning is processing technology using convolutional neural networks (CNNs). Convolutional neural networks are used in a variety of fields, such as image recognition, speech recognition, natural language processing, and machine translation.

[0003] A technology for performing super-resolution using the above-described convolutional neural network is described in Patent Document 1. The image processing device described in Patent Document 1 aims to avoid the so-called gradient vanishing problem (or degradation problem), in which the gradient (interlayer coefficient) becomes zero, by performing the following steps: acquiring an input image; calculating feature residuals by processing the input image using a convolutional layer; performing at least one convolution on the input image; generating output features by applying the feature residuals to the convolved input image; and generating image residuals based on the output features. Applying the image residuals to the input image generates a high-resolution image with a higher resolution than the input image. In other words, so-called super-resolution is achieved. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6276901 Summary of the Invention [Problem to be solved by the invention]

[0005] The image processing device described in Patent Document 1 generates a super-resolution image while avoiding the gradient vanishing problem by adding image residuals to an input image and performing convolution. Conventional image processing techniques extract features (feature points) of a target object from image data captured by a camera or image sensor, and then perform object pose estimation and segmentation. However, to obtain sufficient features, the distance from the object must be set to approximately 1 m. Even when using a particularly high-sensitivity sensor, object estimation is typically performed within a distance of 3 m. Therefore, when estimating an object located farther away than 3 m, the object appears small in the image, dramatically reducing the number of features extracted. As a result, the accuracy of object estimation also decreases dramatically. Furthermore, when the object itself is small, has little texture, or is textureless, the inherently low feature count makes it difficult to perform appropriate learning.

[0006] This invention was conceived in response to the above-mentioned technical challenges, and aims to provide a machine learning device and a machine learning method that can appropriately perform learning (estimation) using a convolutional neural network, while avoiding the loss or disappearance of features (feature points), for objects located far away, small objects, or objects with little texture. [Means for solving the problem]

[0007] In order to achieve the above object, the present invention provides a machine learning device that extracts features of predetermined target data using a convolutional neural network and performs learning (or estimation) on the target data, the device comprising a calculation unit that executes calculation processing using blocks composed of a plurality of layers for performing the learning, the calculation unit including at least a first convolution processing unit that executes calculation processing using a first convolution layer that extracts the features by convolution processing on the target data and differentiates the features from other data, a second convolution processing unit that executes calculation processing using a second convolution layer that increases the amount of information in the channel direction from the target data differentiated by the first convolution processing unit, and a max pooling processing unit that executes calculation processing using a max pooling layer that compresses the target data, the amount of information of which has been increased by the second convolution processing unit, leaving only the maximum value within a predetermined filter size. an upsampling processing unit that performs arithmetic processing by an upsampling layer that expands the size of the target data compressed by the max pooling processing unit and upsamples by interpolating missing parts using neighboring copy values; a third convolution processing unit that performs arithmetic processing by a third convolution layer that converts the target data upsampled by the upsampling processing unit into the target data of a predetermined number of channels; a fourth convolution processing unit that performs arithmetic processing by a fourth convolution layer that converts the target data before the arithmetic processing by the first convolution layer directly into the target data of a predetermined number of channels; and an addition processing unit that adds together the target data converted by the third convolution processing unit and the target data converted by the fourth convolution processing unit.

[0008] Furthermore, the arithmetic unit in the machine learning device of the present invention may further include a batch normalization unit that executes arithmetic processing by a batch normalization layer that performs normalization processing on the target data after the arithmetic processing in the second convolution processing unit, after the arithmetic processing in the max pooling processing unit, and after the arithmetic processing in the third convolution processing unit, respectively.

[0009] In addition, the calculation unit in the machine learning device of the present invention may further include an activation processing unit that performs calculation processing using an activation layer to clarify the features by removing excess data from the target data added together by the addition processing unit.

[0010] The calculation unit in the machine learning device of the present invention performs calculation processing using a plurality of convolution blocks composed of the block and the activation layer, and the machine learning device of the present invention may also include an output layer that outputs the results of the calculation processing using the plurality of convolution blocks.

[0011] On the other hand, the present invention is a machine learning method for extracting features of predetermined target data using a convolutional neural network and performing learning (or estimation) on the target data, wherein arithmetic processing is performed by a block composed of a plurality of layers for performing the learning, and the arithmetic processing by the block includes at least: arithmetic processing by a first convolutional layer that extracts the features by convolutional processing on the target data and differentiates the features from other data (first convolutional processing step); after the arithmetic processing by the first convolutional layer, arithmetic processing by a second convolutional layer that increases the amount of information in the channel direction from the differentiated target data (second convolutional processing step); after the arithmetic processing by the second convolutional layer, arithmetic processing by a max pooling layer that compresses the target data with the increased amount of information, leaving only the maximum value within a predetermined filter size (max pooling processing step); This method is characterized in that, after the arithmetic processing by the despooling layer, an arithmetic processing by an upsampling layer is performed, which expands the size of the compressed target data and upsamples it by interpolating (interpolating) missing parts using surrounding copy values ​​(upsampling processing step); after the arithmetic processing by the upsampling layer, an arithmetic processing by a third convolutional layer is performed, which converts the upsampled target data into target data with a predetermined number of channels (third convolutional processing step);

[0012] Furthermore, the machine learning method of the present invention may be a method in which, after the arithmetic processing by the second convolutional layer, after the arithmetic processing by the upsampling layer, and after the arithmetic processing by the third convolutional layer, arithmetic processing by a batch normalization layer that performs normalization processing on the target data is executed (batch normalization processing step).

[0013] Furthermore, the machine learning method of the present invention may be a method in which, after the calculation processing by the blocks, calculation processing is performed by an activation layer that removes unnecessary data from the added target data to clarify the features (activation processing step).

[0014] The machine learning method of the present invention may be a method of performing arithmetic processing using a plurality of convolutional blocks composed of the block and the activation layer, and outputting the results of the arithmetic processing using the plurality of convolutional blocks. [Effects of the Invention]

[0015] This invention performs learning using a convolutional neural network and performs learning about an object based on the learning results. For example, the position and posture of an object are estimated or predicted from image data (object data) of the object. Alternatively, the motion state and driving state of the object are estimated or predicted from sound data (object data). The machine learning device and machine learning method of this invention perform arithmetic processing using a block in which two layers that expand the features of the object data are inserted into a conventional residual block in machine learning (convolutional neural network). That is, arithmetic processing is performed by adding a max pooling layer and an upsampling layer. The max pooling layer makes it easier to retain features, and the upsampling layer increases the features while restoring the size of the object data. Therefore, the machine learning device and machine learning method of this invention can suppress the reduction of features of the object data and appropriately perform learning about the object data using a block (upsampling residual block) formed by adding the max pooling layer and upsampling layer as described above.

[0016] Therefore, according to the machine learning device and machine learning method of the present invention, for objects such as objects located far away, small objects, or objects with little texture, it is possible to avoid the loss or disappearance of feature quantities (feature points) of the object data related to the object, and to appropriately carry out learning (estimation) using a convolutional neural network. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a machine learning device according to the present invention. [Figure 2] FIG. 2 is a diagram for explaining the problems with conventional technology (machine learning using conventional residual blocks), and is a flowchart (process diagram) showing the content and steps of the computational processing by a residual block that introduces a convolutional layer into the bypass connection. [Figure 3]Figure 3 shows changes in feature amounts (feature points) when object pose estimation is performed using the conventional residual block shown in Figure 2. Figure 3(a) shows an image of feature amounts after arithmetic processing by the second convolutional layer, and Figure 3(b) shows an image of feature amounts after arithmetic processing by the third convolutional layer. [Figure 4] FIG. 4 is a flowchart (process chart) showing the content and steps of basic arithmetic processing by the residual block (upsampling residual block) used in the machine learning device and machine learning method of the present invention. [Figure 5] Fig. 5 is a diagram showing changes in feature amounts (feature points) when object pose estimation is performed using the residual block (upsampling residual block) of the present invention shown in Fig. 4. Fig. 5(a) is a diagram showing an image of feature amounts after arithmetic processing by the second convolutional layer, Fig. 5(b) is a diagram showing an image of feature amounts after arithmetic processing by the max pooling layer, and Fig. 5(c) is a diagram showing an image of feature amounts after arithmetic processing by the upsampling layer. [Figure 6] FIG. 6 is a diagram for explaining another example of the computational processing performed by the machine learning device and machine learning method of the present invention, and is a flowchart (process diagram) showing the content and steps of the computational processing by a residual block (upsampling residual block) to which a batch normalization layer has been added. [Figure 7] Fig. 7 is a diagram showing changes in feature amounts (feature points) when object pose estimation is performed using the residual block (upsampling residual block) of this invention shown in Fig. 6. Fig. 7(a) is a diagram showing an image of feature amounts after arithmetic processing by the first convolutional layer, Fig. 7(b) is a diagram showing an image of feature amounts after arithmetic processing by the second convolutional layer, Fig. 7(c) is a diagram showing an image of feature amounts after arithmetic processing by the batch normalization layer, Fig. 7(d) is a diagram showing an image of feature amounts after arithmetic processing by the max pooling layer, and Fig. 7(e) is a diagram showing an image of feature amounts after arithmetic processing by the upsampling layer. [Figure 8]FIG. 8 is a diagram for explaining another example of the computational processing executed by the machine learning device and machine learning method of the present invention, and is a flowchart showing the computational processing by a plurality of convolutional blocks composed of a residual block (upsampling residual block) with an added batch normalization layer and an activation layer. [Figure 9] FIG. 9 is a flowchart illustrating another example of the computational processing performed by the machine learning device and machine learning method of the present invention, in which an output layer is added to a plurality of convolution blocks each composed of a residual block (upsampling residual block) to which a batch normalization layer has been added and an activation layer (for example, an example of determining a fault in a mechanical device based on sound data). DETAILED DESCRIPTION OF THE INVENTION

[0018] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following embodiments of the present invention will be described with reference to the accompanying drawings. Note that the following embodiments are merely examples of specific embodiments of the present invention and are not intended to limit the scope of the present invention.

[0019] In the machine learning device and machine learning method according to the embodiment of the present invention, a convolutional neural network is used to perform a convolution operation based on a large amount of collected information (target data) to extract features (features, feature points) of the information source. For example, features are extracted from image data (target data) of an object such as a workpiece or a transported object, and the position and orientation of the object are estimated based on the features. Alternatively, features are extracted from sound data (target data) of collected operating sounds of a mechanical device such as an engine or motor, and the operating state or operational status of the mechanical device is estimated or predicted based on the features. To perform such estimation or prediction, the machine learning device and machine learning method according to the embodiment of the present invention executes machine learning using a convolutional neural network using a processor (CPU) 100 of the machine learning device 1 shown in FIG. 1.

[0020] As shown in FIG. 1, the processor 100 of the machine learning device 1 in this embodiment of the present invention mainly includes a detection unit 101, a data acquisition unit 102, a data generation unit 103, a calculation unit 104, a learning unit 105, and an output unit 106.

[0021] The detection unit 101 is a device or apparatus for acquiring various data and information required for executing machine learning using a convolutional neural network by the machine learning device 1, and includes a power supply unit, a microcomputer, a sensor, an input / output interface, etc. The detection unit 101 in the embodiment of the present invention includes, for example, a camera (not shown) that collects image data for image processing, or a microphone (not shown) that collects sound data for sound source estimation and frequency analysis of a mechanical device. The detection unit 101 is electrically connected to the data acquisition unit 102 (described later), and outputs, as detection data, an electrical signal corresponding to the detected values, calculated values, or position information of the various sensors, devices, etc. described above.

[0022] The data acquisition unit 102 acquires predetermined data necessary for generating training data for the convolutional neural network. The various data detected by the detection unit 101 are acquired as necessary as information for generating training data for the convolutional neural network.

[0023] Data generation unit 103 generates input data (learning data) to be used in the convolutional neural network in the embodiment of the present invention from a large amount of data acquired by data acquisition unit 102 or a large amount of data stored in a data storage unit (not shown). This input data or learning data is the "target data" in the embodiment of the present invention, and is used by calculation unit 104 and learning unit 105, which will be described later, to extract "features" and perform learning or estimation of the target.

[0024] The calculation unit 104 mainly performs convolution calculations in a convolutional neural network based on the input data (learning data, target data) generated by the data generation unit 103 as described above. The calculation unit 104 also performs calculation processing using "blocks (upsampling residual blocks)" composed of multiple "layers" for learning or estimating the target object as described above, i.e., for learning about the "target data."

[0025] Specifically, the arithmetic unit 104 in this embodiment of the present invention includes at least a first convolution processing unit that performs arithmetic processing in a "first convolution layer" described below, a second convolution processing unit that performs arithmetic processing in a "second convolution layer" described below, a max pooling processing unit that performs arithmetic processing in a "max pooling layer" described below, an upsampling processing unit that performs arithmetic processing in an "upsampling layer" described below, a third convolution processing unit that performs arithmetic processing in a "third convolution layer" described below, a fourth convolution processing unit that performs arithmetic processing in a "fourth convolution layer" described below, and an addition processing unit (or an adder or addition operator) described below. Details of each "layer" and each "block" used for these arithmetic processing will be described later.

[0026] The learning unit 105 performs learning on the "target data" based on the results of the calculation performed by the calculation unit 104. For example, the learning unit 105 estimates the position and posture of a target object, such as a workpiece or a transported object, from image data (target data) of the target object. Alternatively, the learning unit 105 estimates or predicts the operating state or driving state of a machine device from sound data (target data) of the operating sound of the machine device, such as an engine or motor.

[0027] The output unit 106 executes measures or controls corresponding to the learning results (estimation, prediction, determination, etc.) based on the results of learning performed by the learning unit 105. For example, the output unit 106 outputs the results of estimating the position and posture of an object from image data (object data) as three-dimensional data. Alternatively, the output unit 106 outputs a warning display signal, notification signal, etc. based on the results of estimating or predicting the operating state or driving state of an object based on sound data (object data) (for example, a failure determination result).

[0028] Recently, machine learning using Residual Neural Networks (ResNets) has been used to estimate the pose and predict the state of objects, as described above. ResNets are convolutional neural networks (CNNs) or deep neural networks (DNNs) that use residual blocks. Residual blocks are blocks consisting of residual connections (also known as skip connections, shortcut connections, or bypass connections) that skip the convolutional layer for the input, and addition operators that add the inputs that skipped the convolutional layer. Using residual blocks can improve the processing accuracy of calculations and reduce processing time. However, performing advanced calculations using such residual blocks or residual connections has presented a challenge: the gradient (inter-layer coefficient) becomes zero, which is known as the vanishing gradient problem. Therefore, in the image processing device described in the aforementioned Patent Document 1, for example, as shown in FIG. 2, a convolutional layer (CONV) that performs convolutional operations is introduced in the residual connection (or bypass connection) part (step S4) of the residual block, thereby avoiding the occurrence of the gradient vanishing problem.

[0029] Specifically, in the flowchart (or process diagram) shown in FIG. 2, first, in the convolutional layer (CONV) in step S1, image data is multiplied by a predetermined coefficient to make it easier to extract features (or feature points).

[0030] In the convolutional layer (CONV) in step S2, the features are filtered to leave only the exaggerated features. Figure 3(a) shows an image of the features after the computation in step S2.

[0031] In the convolutional layer (CONV) in step S3, the extracted features are multiplied again by a predetermined coefficient. An image of the features after the calculation process in step S3 is shown in Figure 3(b).

[0032] In the convolution layer (CONV) in step S4, the original image is directly multiplied by a predetermined coefficient. In this step S4, a convolution layer is provided in the bypass connection portion in a conventional general residual block.

[0033] Then, in step S5, the feature values ​​multiplied by the coefficients in step S3 are added to the original image multiplied by the coefficients in step S4. This increases the feature values ​​and decreases values ​​other than the feature values, thereby increasing the difference between them. This makes it possible to avoid or suppress the occurrence of the gradient vanishing problem described above.

[0034] However, even when a convolutional layer is introduced into the bypass connection in the residual block as shown in Figures 2 and 3 above, for example, when performing image processing of a small object located far away, the feature amount may be significantly reduced during the convolution processing process, or the original feature amount may be insufficient. As a result, there is a risk that appropriate learning or accurate estimation may not be possible. Therefore, the machine learning device and machine learning method according to the embodiment of the present invention add a "layer" to the conventional residual block as described above to expand the feature amount of the object (object data) and perform arithmetic processing.

[0035] Specifically, as shown in the flowchart (or process diagram) of FIG. 4, a "max pooling layer" and an "upsampling layer" are incorporated into the conventional residual block shown in FIG. 2 above to perform arithmetic processing. In the flowchart shown in FIG. 4, first, in step S11 (first convolution processing step), the "first convolution layer" (CONV) in the "first convolution processing unit" of the arithmetic unit 104 extracts features (or feature points) by convolution processing of the target object (target data), and differentiates the features from other data. Specifically, similar to step S1 in the flowchart of FIG. 2 above, image data is multiplied by a predetermined coefficient to facilitate feature extraction.

[0036] In step S12 (second convolution processing step), the "second convolution layer" (CONV) in the "second convolution processing unit" of the calculation unit 104 increases the amount of information in the channel direction from the target data differentiated in the "first convolution layer" in step S11. Specifically, similar to step S2 in the flowchart of FIG. 2 above, the features are filtered to leave only exaggerated features. An image of the features after the calculation processing in step S12 is shown in FIG. 5(a).

[0037] In the machine learning device and machine learning method according to the embodiment of the present invention, in the next steps S13 and S14, calculation processes are performed using a "max pooling layer" and an "upsampling layer."

[0038] In step S13 (max pooling processing step), the target data whose information volume has been increased in the "second convolution layer" in step S12 is compressed by the "max pooling layer" in the "max pooling processing unit" of the calculation unit 104, leaving only the maximum value within a predetermined filter size. In other words, max pooling is performed. Specifically, by leaving only the maximum value within the filter for the image data that is the target data, the image size is compressed to half. An image of the feature amounts after the calculation processing in step S13 is shown in Figure 5(b).

[0039] In step S14 (upsampling processing step), the size of the target data compressed in the "max pooling layer" in step S13 is expanded by the "upsampling layer" in the "upsampling processing unit" of the calculation unit 104, and missing parts are interpolated using neighboring copy values. In other words, upsampling is performed. Specifically, the compressed image size is restored and the feature pixels are enlarged. This makes it possible to increase the remaining feature. An image of the feature after the calculation processing in step S14 is shown in Figure 5(c).

[0040] In step S15 (third convolution processing step), the target data upsampled in the "upsampling layer" in step S14 is converted into target data with a predetermined number of channels by the "third convolution layer" (CONV) in the "third convolution processing unit" of the calculation unit 104. Specifically, similar to step S3 in the flowchart of FIG. 2 above, the extracted feature amount is again multiplied by a predetermined coefficient.

[0041] In step S16 (fourth convolution processing step), the target data before the calculation processing by the "first convolution layer" in step S11 is directly converted into target data with a predetermined number of channels by the "fourth convolution layer" (CONV) in the "fourth convolution processing unit" of the calculation unit 104. Specifically, similar to step S4 in the flowchart of FIG. 2 above, the original image is directly multiplied by a predetermined coefficient.

[0042] Then, in step S17 (addition processing step), the "addition processing unit (or adder, addition operator, or addition processing layer)" of the calculation unit 104 adds together the target data converted in the "third convolution layer" in step S15 and the target data converted in the "fourth convolution layer" in step S16. Specifically, similar to step S5 in the flowchart of FIG. 2 above, the feature amount multiplied by the coefficient in step S5 and the original image multiplied by the coefficient in step S6 are added together.

[0043] 4 and 5 above show examples of computational processing by basic "blocks" in the machine learning device and machine learning method according to the embodiment of the present invention. A "block" is configured to perform multiple types of computational processing using multiple "layers." The "block" according to the embodiment of the present invention is composed of the above-mentioned "first convolutional layer," "second convolutional layer," "max pooling layer," "upsampling layer," "third convolutional layer," "fourth convolutional layer," and "summing unit" (summing processing layer or adder). In the following description of the embodiment of the present invention, the "block" composed of each "layer" and "summing unit" shown in the flowchart (or process diagram) of FIG. 4 is referred to as an "upsampling residual block" (UR block).

[0044] In this way, the machine learning device and machine learning method according to the embodiment of the present invention can reduce unnecessary data and expand the area in which features remain compared to conventional techniques by performing arithmetic processing using an "upsampling residual block" as shown in the flowchart in Figure 4. This makes it possible to suppress feature loss and execute learning appropriately and efficiently.

[0045] 6 to 9 show other examples of the arithmetic processing executed by the machine learning device and machine learning method according to the embodiment of the present invention. In the example shown in Fig. 6, a step of performing arithmetic processing using a "batch normalization layer" (batch normalization processing step) is added to the example of the basic arithmetic processing shown in Fig. 4 above.

[0046] Specifically, as shown in the flowchart (or process diagram) of Fig. 6, a "batch normalization layer" is incorporated into the basic "upsampling residual block" shown in Fig. 4, and calculation processing is performed. In the embodiment of the "upsampling residual block" shown in the flowchart of Fig. 6, steps (processes) having the same calculation processing content as the steps (processes) of the basic "upsampling residual block" shown in the flowchart of Fig. 4 are assigned the same step numbers (process numbers) as in the flowchart of Fig. 4.

[0047] In the flowchart (or process diagram) shown in FIG. 6, in step S21, a "batch normalization layer" (BN) in the "batch normalization processing unit" of the calculation unit 104 performs normalization processing on the target data that has been calculated in the "first convolution layer" in step S11. The normalization processing in this case is a calculation processing equivalent to conventional batch normalization. By performing this batch normalization processing, it is possible to speed up the convergence of learning and stabilize learning.

[0048] Similarly, in step S22, the "batch normalization layer" in the "batch normalization processing unit" of the calculation unit 104 performs normalization processing on the target data arithmetically processed in the "second convolutional layer" in step S12. Images of features after the arithmetic processing in step S11, the arithmetic processing in step S12, and the arithmetic processing in step S22 are shown in (a), (b), and (c) of FIG. 7, respectively. (b) of FIG. 7 shows an image of so-called fired features, which are the result of performing arithmetic processing in the "second convolutional layer" on the features after the arithmetic processing in the "first convolutional layer" shown in (a) of FIG. 7. (c) of FIG. 7 shows an image of features whose scale has changed as a result of performing arithmetic processing in the "batch normalization layer" in step S22.

[0049] Then, in step S23, the "batch normalization layer" (BN) in the "batch normalization processing unit" of the calculation unit 104 performs normalization processing on the target data calculated in the "upsampling layer" in step S13 and the "upsampling layer" in step S14. Images of the features after the calculation processing in step S13 and the feature values ​​after the calculation processing in step S14 are shown in (d) and (e) of FIG. 7, respectively. (d) of FIG. 7 shows an image of compressed features as a result of the calculation processing by the "max pooling layer" in step S13. (e) of FIG. 7 shows an image of expanded pixels as a result of the calculation processing by the "upsampling layer" in step S14. As can be seen from (e) of FIG. 7, the calculation processing by the "upsampling layer" in step S14 in the flowchart of FIG. 6 increases the area of ​​features with characteristic values. This prevents feature loss and enables appropriate and efficient learning.

[0050] In the calculation process by the "upsampling layer" in the embodiment of the present invention, the bilinear method, nearest method, or nearest neighbor method can be used as a resampling method. Comparing the two, the bilinear method has the advantage that the expanded values ​​change, making it less likely that feature values ​​will be erased all at once in the subsequent calculation process.

[0051] In the example shown in Figure 8, a process of performing arithmetic processing using multiple convolution blocks including an "activation layer" (activation processing step) is added to the "upsampling residual block" shown in Figure 6 above.

[0052] Specifically, as shown in the central flowchart (or process diagram) of FIG. 8, the arithmetic processing is repeated multiple times by the "upsampling residual block" (UR block) to which the "batch normalization layer" shown in FIG. 6 above has been added. At the same time, the arithmetic processing is performed by weaving an "activation layer" (ReLU) between the "upsampling residual block" and the next "upsampling residual block." In the example of the "upsampling residual block" shown in the flowchart on the right side of FIG. 8, steps (processes) having the same arithmetic processing content as the steps (processes) of the "upsampling residual block" shown in the flowchart of FIG. 6 above are assigned the same step numbers (process numbers) as in the flowchart of FIG. 6.

[0053] In the flowchart (or process chart) shown in the center of Fig. 8, steps S31, S33, and S35 each represent one "upsampling residual block" (UR block) made up of the "layers" of steps S11 to S17 and steps S21 to S23 shown on the right side of Fig. 8. Therefore, in the embodiment shown in Fig. 8, the arithmetic processing by the "upsampling residual block" is repeated three times between step S31 and step S36, which will be described later.

[0054] In the machine learning device and machine learning method according to the embodiment of the present invention, an "activation layer" (ReLU) is used for calculation between the "upsampling residual blocks" in steps S31, S33, and S35. That is, in steps S32, S34, and S36, the "activation layer" in the "activation processing unit" of the calculation unit 104 performs activation processing on the target data calculated by the "upsampling residual blocks" in steps S31, S33, and S35. The "activation layer" is a layer that removes unnecessary data from the target data added by the "addition processing unit" in the "upsampling residual blocks" to clarify the features. For example, the activation processing is performed using a "rectified linear unit" (ReLU), which is useful for solving the aforementioned gradient vanishing problem, as the activation function.

[0055] Furthermore, in the machine learning device and machine learning method according to the embodiment of the present invention, in the flowchart (or process chart) shown on the left side of Fig. 8, steps S41, S42, S43, and S44 each represent a single "convolution block" (COV block) composed of the "upsampling residual blocks" of steps S31, S33, and S35, respectively, and the "activation layers" of steps S32, S34, and S36, respectively, shown in the center of Fig. 8. Therefore, in the example shown in Fig. 8, the arithmetic processing by the "convolution block" described above is repeated four times between steps S41 and S44.

[0056] In the example shown in FIG. 9, an "output layer" (Output Head) is provided at the end of the multiple "convolution blocks" shown in FIG. 8 above.

[0057] Specifically, in the flowchart (or process diagram) shown in FIG. 9, an "output layer" (Output Head) is added in step S51 after the combination of four "convolution blocks" shown in steps S41 to S44. The "output layer" (or output unit, output device, etc.) in the embodiment of the present invention outputs the results of arithmetic processing by the above-mentioned multiple "convolution blocks." For example, in the image processing described above, learning about an object is performed from image data (target data), and based on the results of the learning, the position and orientation of the object is estimated and output as three-dimensional data. Alternatively, learning about a mechanical device is performed from sound data (target data) such as the operating sounds of the mechanical device, and based on the results of the learning, the operating state or operating condition of the mechanical device is estimated and a fault is determined. Alternatively, the machine life, etc. is predicted. Based on the results of the learning, a warning display signal, a notification signal, etc. are output.

[0058] As described above, in the machine learning device and machine learning method according to the embodiment of the present invention, a "max pooling layer" and an "upsampling layer" are added to a conventional residual block in machine learning (convolutional neural network) to expand the features of target data, and arithmetic processing is performed using these "max pooling layer" and "upsampling layer." The "max pooling layer" makes it easier to retain features, and the "upsampling layer" increases the features while restoring the size of the target data. Therefore, in the machine learning device and machine learning method according to the embodiment of the present invention, the "upsampling residual block" constructed by adding the above-described "max pooling layer" and "upsampling layer" suppresses a decrease in the features of the target data, and allows appropriate learning of the target data to be performed.

[0059] Therefore, according to the machine learning device and machine learning method of the embodiment of the present invention, for example, for a target object such as an object located far away, a small object, or an object with little texture, it is possible to avoid the loss or disappearance of features (feature points) of the target data related to the target object, and to appropriately perform learning (estimation) using a convolutional neural network. [Explanation of symbols]

[0060] 1. Machine learning device 100 processors (CPUs) 101 Detection unit 102 Data Acquisition Unit 103 Data Generation Unit 104 Arithmetic section 105 Learning Department 106 Output section

Claims

1. A machine learning device that extracts features of predetermined target data using a convolutional neural network and performs learning on the target data, a calculation unit that executes calculation processing using blocks composed of a plurality of layers for performing the learning; The calculation unit at least, a first convolution processing unit that extracts the feature amount by convolution processing on the target data and performs arithmetic processing by a first convolution layer that separates and differentiates the feature amount from other data; a second convolution processing unit that performs a calculation process using a second convolution layer to increase the amount of information in the channel direction from the differentiated target data; a max pooling processing unit that performs a calculation process using a max pooling layer that compresses the target data with the increased amount of information by leaving only the maximum value within a predetermined filter size; an upsampling processing unit that expands the size of the compressed target data and performs arithmetic processing using an upsampling layer that upsamples missing parts using neighboring copy values; a third convolution processing unit that performs a calculation process using a third convolution layer to convert the upsampled target data into the target data with a predetermined number of channels; a fourth convolution processing unit that performs a calculation process by a fourth convolution layer that directly converts the target data before the calculation process by the first convolution layer into the target data having a predetermined number of channels; an addition processing unit that adds together the target data converted by the third convolution processing unit and the target data converted by the fourth convolution processing unit. A machine learning device characterized by:

2. The machine learning device according to claim 1 , The calculation unit The apparatus further includes a batch normalization processing unit that executes a calculation process by a batch normalization layer that performs a normalization process on the target data after the calculation process in the second convolution processing unit, after the calculation process in the max pooling processing unit, and after the calculation process in the third convolution processing unit, respectively. A machine learning device characterized by:

3. The machine learning device according to claim 1 or 2, The calculation unit The method further includes an activation processing unit that performs calculation processing using an activation layer to clarify the feature amount by deleting unnecessary data from the target data added by the addition processing unit. A machine learning device characterized by:

4. The machine learning device according to claim 3, The calculation unit Executes arithmetic processing by a plurality of convolution blocks configured by the block and the activation layer, and and an output layer that outputs the results of the arithmetic processing by the plurality of convolution blocks. A machine learning device characterized by:

5. A machine learning method for extracting features of predetermined target data using a convolutional neural network and performing learning on the target data, comprising: Executes arithmetic processing using blocks composed of multiple layers for performing the learning, and The calculation process by the block is as follows: at least, extracting the feature amount by convolution processing on the target data, and performing a calculation process by a first convolution layer that separates and differentiates the feature amount from other data; After the calculation process by the first convolution layer, a calculation process by a second convolution layer is performed to increase the amount of information in a channel direction from the differentiated target data; After the arithmetic processing by the second convolutional layer, a arithmetic processing by a max pooling layer is performed to compress the target data with the increased amount of information by leaving only the maximum value within a predetermined filter size; After the calculation process by the max pooling layer, a calculation process is performed by an upsampling layer that expands the size of the compressed target data and upsamples the missing part using neighboring copy values; After the arithmetic processing by the upsampling layer, a arithmetic processing by a third convolution layer is performed to convert the upsampled target data into the target data with a predetermined number of channels; Execute a calculation process by a fourth convolution layer that directly converts the target data before the calculation process by the first convolution layer into the target data with a predetermined number of channels; The target data transformed by the arithmetic processing by the third convolutional layer and the target data transformed by the arithmetic processing by the fourth convolutional layer are added together. A machine learning method characterized by:

6. The machine learning method according to claim 5, After the arithmetic processing by the second convolution layer, after the arithmetic processing by the upsampling layer, and after the arithmetic processing by the third convolution layer, arithmetic processing by a batch normalization layer that performs normalization processing on the target data is performed. A machine learning method characterized by:

7. 7. The machine learning method according to claim 5 or 6, After the calculation process by the block, calculation process by an activation layer is performed to clarify the feature by deleting unnecessary data from the added target data. A machine learning method characterized by:

8. The machine learning method according to claim 7, Executes arithmetic processing by a plurality of convolution blocks configured by the block and the activation layer, and Outputting the results of the arithmetic processing by the plurality of convolution blocks A machine learning method characterized by:

Images