Information processing apparatus, and information processing method

Abstract

To provide a technique to promote integration of a local feature and an entire feature.SOLUTION: An information processing apparatus comprises: a first operation unit that performs first conversion that is a plurality of stages of local feature conversion; and a second operation unit that performs second conversion that is feature conversion in a wider area than the first conversion. The second conversion is feature conversion that changes the number of elements and the number of dimensions of a feature quantity in increase and decrease directions different from the first conversion.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to feature transformation techniques. [Background technology]

[0002] There are convolutional neural networks (CNNs) that perform feature transformation by repeating local processing that shares weights. Another example is the Vision Transformer (ViT), which divides input data into multiple regions, examines the relevance between the regions, and determines which region's features to extract based on the relevance (Non-Patent Document 1). CNNs and ViTs can gradually integrate local features and identify patterns by repeating feature transformation in units of local regions in multiple layers.

[0003] Furthermore, to promote the integration of the overall features of the input data with local features, methods known as the Feature Pyramid Network (Non-Patent Document 2) and the Stacked Hourglass Network (Non-Patent Document 3), which sequentially and stepwise repeat aggregation (convolution) and decomposition (deconvolution processing and upsampling processing), are widely used. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Dosovitskiy, Alexey, et al. “An image is worth 16x16 words: Transformers for image recognition at scale.” arXiv preprint arXiv:2010.11929 (2020). [Non-patent document 2] Lin, Tsung-Yi, et al. “Feature pyramid networks for object detection.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. [Non-patent document 3] Newell, Alejandro, Kaiyu Yang, and Jia Deng. “Stacked hourglass networks for human pose estimation.” Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14. Springer International Publishing, 2016. Summary of the Invention [Problem to be solved by the invention]

[0005] Methods such as the Feature Pyramid Network and Stacked Hourglass Network constantly maintain the spatial arrangement of features and have a small degree of freedom in the transformations at each stage, making them less susceptible to overfitting. However, the large number of transformation stages results in poor processing efficiency. On the other hand, methods using fully connected layers (MLP, Multilayer Perceptron) perform batch transformations without distinguishing between local and global, resulting in good processing efficiency. However, the high degree of freedom in the transformations makes them prone to overfitting, and the desired inference results may not be obtained. This invention provides technology for promoting the integration of local and global features. [Means for solving the problem]

[0006] One aspect of the present invention is characterized in that it comprises a first calculation unit that performs a first transformation, which is a local multi-stage feature transformation, and a second calculation unit that performs a second transformation, which is a feature transformation that is broader than the first transformation, and the second transformation is a feature transformation that changes the number of elements and the number of dimensions of a feature in a direction that is different from that of the first transformation. [Effects of the Invention]

[0007] According to the present invention, it is possible to promote the integration of local features and global features. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of the hardware configuration of an information processing device that performs arithmetic processing in a neural network. [Figure 2] FIG. 1 is a block diagram showing an example of the functional configuration of an information processing device that operates as an inference device. [Figure 3] FIG. 2 is a block diagram showing an example of the configuration of a calculation unit 202. [Figure 4] FIG. 1 is a diagram showing a specific example of a neural network. [Figure 5] FIG. 1 is a diagram showing the processing steps in a neural network. [Figure 6] 4A and 4B are diagrams showing a specific example of processing in a reconstruction unit 303. [Figure 7] FIG. 1 is a diagram showing a specific example of a neural network. [Figure 8] FIG. 1 is a diagram showing the processing steps in a neural network. [Figure 9] FIG. 1 is a diagram showing a specific example of a neural network. [Figure 10A] FIG. 4 is a diagram showing a method for acquiring intermediate features of the ResNet unit 401. [Figure 10B] FIG. 4 is a diagram showing a method for acquiring intermediate features of the ResNet unit 401. [Figure 11] FIG. 2 is a diagram showing an example of the configuration of a calculation unit 202. [Figure 12] FIG. 1 is a diagram showing a specific example of a neural network. [Figure 13]Flowchart of the processing steps in a neural network. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] [First embodiment] The neural network used in this embodiment is a hierarchical neural network that has a layer that gradually aggregates features while maintaining spatial positional relationships, like a conventional convolutional layer, and a layer that re-expands the features in the spatial direction using a fully connected layer.With this configuration, the neural network according to this embodiment can perform conversion processing efficiently.

[0011] First, an example of the hardware configuration of an information processing device that performs arithmetic processing in a neural network according to this embodiment will be described using the block diagram of Fig. 1. Computer devices such as PCs, smartphones, and tablet terminal devices can be applied to the information processing device according to this embodiment. Note that the hardware configuration shown in Fig. 1 is an example of a hardware configuration that can be applied to an information processing device, and can be modified / altered as appropriate.

[0012] The control device 101 is a processor such as a CPU (Central Processing Unit), and executes various processes using computer programs and data stored in the RAM 104. As a result, the control device 101 controls the operation of the entire information processing device, and also executes or controls various processes that will be described as processes performed by the information processing device.

[0013] The arithmetic device 102 can be implemented using a GPU (Graphics Processing Unit) or other calculation processing circuits, and executes various types of calculation processing under the control of the control device 101.

[0014] A ROM (Read Only Memory) 103 stores setting data for the information processing device, computer programs and data relating to the startup of the information processing device, computer programs and data relating to the basic operation of the information processing device, and the like.

[0015] The RAM 104 (Random Access Memory) has an area for storing computer programs and data loaded from the ROM 103 or the external storage device 105. The RAM 104 also has an area for storing computer programs and data received from the outside via the communication unit 108. The RAM 104 also has a work area used when the control device 101 and the arithmetic device 102 execute various processes. In this way, the RAM 104 can provide various areas as needed.

[0016] The external storage device 105 is a large-capacity information storage device such as a hard disk drive, etc. The external storage device 105 stores an OS (Operating System), computer programs and data for causing the control device 101 and the arithmetic device 102 to execute or control various processes described as processes performed by the information processing device, and the like.

[0017] The external storage device 105 may include an optical disk such as a flexible disk (FD) or compact disk (CD) that is detachable from the information processing device, a magnetic or optical card, an IC card, a memory card, or the like.

[0018] The input unit 106 is a user interface such as a keyboard, a mouse, a dial, a touch panel screen, etc., which can be operated by a user to input various instructions and information to the information processing device. The input unit 106 may also include various sensors.

[0019] The display unit 107 has a liquid crystal screen or a touch panel screen, and can display the processing results of the control device 101 and the arithmetic device 102 as images, characters, etc. The display unit 107 may be a projection device such as a projector that projects images and characters.

[0020] The communication unit 108 performs data communication with the outside via a network such as a LAN or the Internet. The control device 101, the arithmetic unit 102, the ROM 103, the RAM 104, the external storage device 105, the input unit 106, the display unit 107, and the communication unit 108 are all connected to a system bus 109.

[0021] Next, an example of the functional configuration of an information processing device that operates as an inference device for performing a face recognition task, which is a task of determining whether a face of a person included in one image and a face of a person included in another image are the faces of the same person, will be described using the block diagram of FIG. 2. In this embodiment, a case will be described in which each functional unit in FIG. 2 is implemented by software (computer program). In the following, each functional unit in FIG. 2 will be described as the subject of processing, but in reality, the function of the functional unit is realized by the control device 101 or the arithmetic device 102 executing a computer program corresponding to the functional unit. Note that some or all of the functional units in FIG. 2 may be implemented by hardware.

[0022] The acquisition unit 201 acquires an image including a person's face. The method of acquiring the image by the acquisition unit 201 is not limited to a specific acquisition method. For example, the acquisition unit 201 may acquire an image stored in the external storage device 105, may acquire an image captured by the input unit 106 as an imaging device, or may acquire an image received from outside via the communication unit 108. Furthermore, the acquisition unit 201 may perform face detection processing on an image including a person's face to identify the area of the face in the image, and acquire an image within the area.

[0023] The calculation unit 202 calculates facial feature amounts, which are feature amounts of a person's face, from the image acquired by the acquisition unit 201. In this embodiment, the calculation unit 202 inputs the image acquired by the acquisition unit 201 into a neural network and performs arithmetic processing of the neural network to calculate facial feature amounts, which are feature amounts of the person's face in the image. In this embodiment, a convolutional neural network (CNN) is used as the neural network. As the structure of the CNN, ResNet, etc. introduced in K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In ECCV, 2016, etc. may be used. Alternatively, a neural network known as Vision Transformer (ViT) and described in Alexey Dosovitskiy, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. may be used. The configuration of the neural network is not limited to these, but the neural network is configured to internally incorporate the method described below.

[0024] The matching unit 203 matches the facial feature amount calculated by the calculation unit 202 for one image acquired by the acquisition unit 201 with the facial feature amount calculated by the calculation unit 202 for the other image acquired by the acquisition unit 201. For example, if the similarity between the matched facial feature amounts is equal to or greater than a threshold, the matching unit 203 determines that the face of a person in one image and the face of a person in the other image are the face of the same person. On the other hand, if the similarity between the matched facial feature amounts is less than the threshold, the matching unit 203 determines that the face of a person in one image and the face of a person in the other image are not the face of the same person. The matching unit 203 then outputs the result of this determination as the result of face authentication. The output format of the result of face authentication is not limited to a specific output format. For example, the matching unit 203 may display the result of face authentication using an image or text on the display unit 107, or may transmit the result of face authentication to an external device via the communication unit 108.

[0025] Next, an example of the configuration of the calculation unit 202 will be described using the block diagram in Fig. 3. The acquisition unit 301 applies preprocessing to the image acquired by the acquisition unit 201, and generates an image as a facial image in a format that can be input to the conversion unit 302 at the subsequent stage. For example, the acquisition unit 301 identifies a human face area from the image acquired by the acquisition unit 201 using well-known techniques. The acquisition unit 301 then performs processes such as cropping, resizing, and normalization on the image within that area to match the input format of the conversion unit 302 at the subsequent stage, and generates an image to be input to the conversion unit 302 as a facial image.

[0026] The transformation unit 302 applies feature transformation to the facial image generated by the acquisition unit 301 by inputting the facial image to a neural network and performing arithmetic processing in the neural network, thereby calculating facial feature amounts, which are facial feature amounts of the person in the facial image, as a result of the arithmetic processing. The neural network used by the transformation unit 302 to calculate the facial feature amounts may be, for example, a convolutional neural network (CNN). The CNN may have a structure such as ResNet, which is introduced in K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In ECCV, 2016, etc. Alternatively, the neural network used by the transformation unit 302 to calculate the facial feature amounts may be, for example, a neural network known as VisionTransformer (ViT) and described in Alexey Dosovitskiy, et al., An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.

[0027] In the conventional method, matching is performed using facial feature amounts acquired by the conversion unit 302. In contrast, in this embodiment, a reconstruction unit 303 and a conversion unit 304 are applied to improve the efficiency of feature conversion.

[0028] The reconstruction unit 303 applies processing including matrix rearrangement to the facial feature amounts calculated by the conversion unit 302, thereby optimizing the facial feature amounts without being restricted by spatial positions. The conversion unit 304 applies feature conversion processing to the facial feature amounts by inputting the facial feature amounts optimized by the reconstruction unit 303 into a neural network and performing arithmetic processing on the neural network. As with the conversion unit 302, the conversion unit 304 can use a neural network such as ResNet or ViT as the neural network. However, the neural network used by the conversion unit 304 does not need to have the same structure as the neural network used by the conversion unit 302.

[0029] The matching unit 203 matches the facial feature amount obtained by the conversion unit 304 for one image acquired by the acquisition unit 201 with the facial feature amount obtained by the conversion unit 304 for the other image acquired by the acquisition unit 201.

[0030] Next, a specific example of the neural network according to this embodiment will be described with reference to Fig. 4. The conversion unit 302 inputs a face image to a ResNet 401 and performs arithmetic processing of the ResNet 401 to calculate face feature amounts.

[0031] The reconstruction unit 303 inputs the facial feature amount calculated using the ResNet 401 to the MLP 402 that performs linear transformation processing, and performs calculation processing of the MLP 402 to transform the facial feature amount.

[0032] Furthermore, the reconstruction unit 303 converts the facial feature amounts converted by the MLP 402 by inputting the facial feature amounts converted by the MLP 402 to a Reshape 403 that performs a shape transformation process on a matrix and performing arithmetic processing on the Reshape 403. The conversion unit 304 converts the facial feature amounts by inputting the facial feature amounts converted by the Reshape 403 to a ResNet 404 and performing arithmetic processing on the ResNet 404.

[0033] The matching unit 203 matches the facial feature amount obtained by the ResNet 404 for one image acquired by the acquisition unit 201 with the facial feature amount obtained by the ResNet 404 for the other image acquired by the acquisition unit 201.

[0034] The structure of ResNet401 and the structure of ResNet404 may or may not be the same. For example, the scale of each of ResNet401 and ResNet404 may be adjusted individually, taking into account computational resources. For example, the number of layers of ResNet401 may be greater than the number of layers of ResNet404. Furthermore, for example, both or one of ResNet401 and ResNet404 may be replaced with ViT.

[0035] Next, the processing steps in the neural network according to this embodiment will be described with reference to FIG. 5. The transformer 302 inputs a facial image 501 to the ResNet 401 and performs computational processing (feature transformation) on the ResNet 401. As a result, the transformer 302 gradually reduces the resolution of the facial features during the feature transformation process, similar to a typical neural network for acquiring features, and ultimately calculates one-dimensional tensor features 502 as the facial features. To improve the processing accuracy of a trained neural network with this structure, further feature transformation of the one-dimensional tensor features 502 can be considered. For example, transformation using a fully connected layer (MLP) is applicable. However, feature transformation using an MLP does not have the same constraints as the weight sharing of a convolutional neural network. This provides superior approximation performance for arbitrary transformations, but it is prone to overfitting.

[0036] Therefore, in this embodiment, a reconstruction unit 303 is introduced to apply a neural network such as ResNet, which is less likely to cause overlearning and has good processing efficiency, to the one-dimensional tensor feature 502.

[0037] The reconstruction unit 303 inputs the one-dimensional tensor feature 502 to the MLP 402 and performs calculation processing (linear transformation processing) of the MLP 402 to calculate the one-dimensional tensor feature 503 in which the number of elements in the one-dimensional tensor feature 502 is increased.

[0038] Next, the reconstruction unit 303 inputs the one-dimensional tensor feature 503 to Reshape 403 and performs calculation processing of Reshape 403. As a result, the reconstruction unit 303 shapes the one-dimensional tensor feature 503 and calculates "three-dimensional facial feature 504, which is a three-dimensional facial feature" by increasing the number of dimensions of the one-dimensional tensor feature 503.

[0039] The conversion unit 304 inputs the three-dimensional face feature amount 504 to the ResNet 404 and performs calculation processing of the ResNet 404 to calculate a final output feature amount 505, which is a one-dimensional tensor feature amount.

[0040] The matching unit 203 matches the final output feature 505 obtained by the ResNet 404 for one image acquired by the acquisition unit 201 with the final output feature 505 obtained by the ResNet 404 for the other image acquired by the acquisition unit 201.

[0041] Next, a specific example of processing in the reconstruction unit 303 will be described with reference to Fig. 6. Fig. 6 shows processing by the reconstruction unit 303 to convert a one-dimensional tensor feature amount having four elements into a three-dimensional facial feature amount expressed as a 4 x 4 x 1 matrix. The conversion unit 302 calculates a one-dimensional tensor feature amount 601 having four elements from the facial image.

[0042] The reconstruction unit 303 calculates the feature quantity 602 of a one-dimensional tensor having 16 elements by inputting the feature quantity 601 of the one-dimensional tensor to the MLP 402 and performing arithmetic processing (linear transformation processing) of the MLP 402. The feature quantity 602 of the one-dimensional tensor having 16 elements is calculated by applying individual linear transformation parameters to each element of the feature quantity 601 of the one-dimensional tensor having four elements.

[0043] The reconstruction unit 303 then inputs the one-dimensional tensor feature 602 to Reshape 403 and performs arithmetic processing of Reshape 403 to rearrange the elements of the one-dimensional tensor feature 602 and calculate a 4×4×1 three-dimensional face feature 603. Fig. 6 shows a case in which the first four elements of the one-dimensional tensor feature 602 are simply extracted and arranged in the column direction to calculate the 4×4×1 three-dimensional face feature 603.

[0044] In this way, the reconstruction unit 303 converts the feature quantity of a one-dimensional tensor into a three-dimensional feature quantity, which is a feature quantity in a format that can be input to the conversion unit 304. Note that, although Fig. 6 shows an example in which the feature quantity of a one-dimensional tensor having four elements is converted into a feature quantity expressed as a 4 x 4 x 1 matrix, the data format is not limited to this, and the reconstruction unit 303 can perform any shape conversion in accordance with the data formats of the conversion units 302 and 304.

[0045] The processing steps in the neural network according to this embodiment will be described with reference to the flowchart in Fig. 13. The details of the processing in each step in Fig. 13 are as described above.

[0046] In step S1301, the transformation unit 302 performs feature transformation on the face image 501 to calculate one-dimensional tensor feature amounts 502 as face feature amounts. In step S1302, the reconstruction unit 303 calculates one-dimensional tensor feature amounts 503 by increasing the number of elements in the one-dimensional tensor feature amounts 502.

[0047] In step S1303, the reconstruction unit 303 calculates three-dimensional facial feature amounts 504 by increasing the number of dimensions of the one-dimensional tensor feature amounts 503. In step S1304, the transformation unit 304 performs feature transformation on the three-dimensional facial feature amounts 504 to calculate final output feature amounts 505.

[0048] Next, the inference and learning methods in the neural network of this embodiment will be described again with reference to Figure 5. In the conventional method, during inference, feature transformation is performed on a face image 501 using only the transformation unit 302 to obtain one-dimensional tensor feature amounts 502, and the one-dimensional tensor feature amounts 502 calculated for one image are compared with the one-dimensional tensor feature amounts 502 calculated for the other image. Furthermore, during learning, loss is calculated using the one-dimensional tensor feature amounts 502.

[0049] On the other hand, in this embodiment, a reconstruction unit 303 and a conversion unit 304 are further added to the conversion unit 302 (ResNet401) that has been pre-trained by the conventional learning method described above. Here, the combination of the conversion unit 302, the reconstruction unit 303, and the conversion unit 304 is treated as a new feature extraction model.

[0050] During inference using the neural network according to this embodiment, the information processing device performs feature transformation on the facial image 501 using the transformation unit 302, the reconstruction unit 303, and the transformation unit 304 to calculate the final output feature 505, and performs matching as described above using the final output feature 505.

[0051] During training of the neural network according to this embodiment, the information processing device calculates the error (loss) between the final output feature 505 and the feature, which is training data corresponding to the facial image 501, and updates the neural network parameters to reduce this error. This error can be backpropagated to the input layer of the conversion unit 302. However, in this embodiment, the parameters of the pre-trained conversion unit 302 (ResNet 401) are not updated, and the parameters of the reconstruction unit 303 (MLP 402 and Reshape 403) and the conversion unit 304 (ResNet 404) are trained. This enables faster training than training all parameters at once. Furthermore, because the output of the conversion unit 302 is further subjected to feature transformation, the neural network processing accuracy is higher than when inference is performed by the conversion unit 302 alone. However, the training method is not limited to this, and training may be performed including the conversion unit 302.

[0052] Finally, we will explain the effects obtained by the above processing procedure. In conventional methods, typical neural networks gradually reduce the resolution of features during the feature transformation process, ultimately obtaining one-dimensional tensor features. To improve the processing accuracy of a trained neural network with this structure, further feature transformation of one-dimensional tensor features can be considered. For example, transformation using a fully connected layer (MLP) is applicable, but this is prone to overfitting.

[0053] On the other hand, in this embodiment, the reconstruction unit 303 transforms the shape of the one-dimensional tensor feature, making it possible to apply a neural network such as ResNet again. Generally, compared to neural networks consisting only of fully connected layers, neural networks such as ResNet are less likely to overfit even when the number of layers and parameters is increased. Therefore, it is possible to add a larger-scale neural network than before to a trained model, thereby improving the accuracy of neural network processing.

[0054] As described with reference to FIG. 6 , the reconstruction unit 303 may increase the number of dimensions of a one-dimensional tensor feature by linear transformation with independent parameters. When increasing the number of dimensions of a feature, conventional methods such as a feature pyramid network or a stacked hourglass network perform redundant processing. Specifically, this method involves deconvolution, which simply concatenates zero elements to increase the feature. On the other hand, in this embodiment, a linear transformation with independent parameters is applied, and therefore the processing efficiency of the neural network is higher than that of redundant processing such as deconvolution.

[0055] Furthermore, general feature transformation preserves the spatial relationship of features, meaning that the spatial arrangement of each element of a feature is not disrupted before and after feature transformation. For example, the element in the upper left corner of a feature consistently corresponds to the upper left region of the input image, no matter how many times feature transformation is performed. While this constraint facilitates learning, it only allows extraction of features related to spatial relationships. In contrast, the reconstruction unit 303 performs processing to swap arbitrary matrix elements, allowing it to extract features that would be missed by feature transformation processing that restricts spatial relationships. At the same time, because the reconstruction unit 303 has learning parameters, it is expected to optimize features so that the downstream transformation unit 304 can efficiently perform feature transformation, in combination with the above-mentioned processing to swap arbitrary matrix elements.

[0056] In this way, in this embodiment, a neural network with high processing accuracy and efficiency can be applied as an additional feature conversion unit. Furthermore, by fixing the weights of the conversion unit 302 (ResNet401) and training only the conversion unit 304 (ResNet404), it is possible to maintain the tendency of the output of the conversion unit 302.

[0057] [Second embodiment] In the following embodiments, including this embodiment, differences from the first embodiment will be described, and unless otherwise specified below, they will be considered to be the same as the first embodiment. In the first embodiment, a case was shown in which the conversion unit 302 and the conversion unit 304 use ResNet. Also, in the first embodiment, the reconstruction unit 303 receives only the output from the conversion unit 302 (the one-dimensional tensor feature 502) as input. In contrast, a specific example of a neural network according to this embodiment will be described with reference to FIG. 7.

[0058] In this embodiment, the conversion unit 302 and the conversion unit 304 use ViT. Also, in this embodiment, the reconstruction unit 303 receives the facial image 501 in addition to the output from the conversion unit 302 (the one-dimensional tensor feature amount 502).

[0059] In addition, in this embodiment, a case will be described in which an information processing device executes an arbitrary object detection task, which is a task for detecting an object from an image. The input of the arbitrary object detection task is an image including an object to be detected. The output of the arbitrary object detection task is the position (detection coordinates) of a frame (detection frame) surrounding the object detected from the image, and the size (vertical and horizontal dimensions) of the detection frame.

[0060] The transformation unit 302 calculates facial feature amounts by inputting the facial image generated by the acquisition unit 301 to ViT 701 and performing arithmetic processing of ViT 701. The reconstruction unit 303 converts the facial feature amounts calculated using ViT 701 by inputting the facial feature amounts calculated using ViT 701 to MLP 402 and performing arithmetic processing of MLP 402. The reconstruction unit 303 then converts the facial feature amounts converted by MLP 402 by inputting the facial feature amounts converted by MLP 402 to Reshape 403 and performing arithmetic processing of Reshape 403. The reconstruction unit 303 then inputs the facial feature amounts converted by Reshape 403 and the facial image generated by the acquisition unit 301 to concatenate 704 and performs arithmetic processing of concatenate 704. The reconstruction unit 303 acquires the facial image generated by the acquisition unit 301 by skip joining (shortcut joining). As a result, the reconstruction unit 303 generates concatenated information that concatenates the facial feature converted by Reshape 403 and the facial image generated by the acquisition unit 301 as an "optimized feature." Note that processing other than concatenate may be applied. For example, element product or addition of the facial feature and the facial image may be applied. In other words, as long as a feature obtained by fusing the facial feature and the facial image can be generated as an "optimized feature," various fusion methods such as concatenation, element product, and addition may be applied. The conversion unit 304 inputs the feature generated by concatenate 704 into ViT 705 and performs arithmetic processing of ViT 705 to convert the feature.

[0061] The structure of ViT701 and the structure of ViT705 may or may not be the same. For example, the scale of ViT701 and ViT705 may be adjusted individually in consideration of computational resources. Also, for example, both or one of ViT701 and ViT705 may be replaced with ResNet.

[0062] Next, the processing steps in the neural network according to this embodiment will be described with reference to Fig. 8. The conversion unit 302 inputs the face image 501 to ViT 701 and performs calculation processing (feature conversion) of the ViT 701 to calculate a one-dimensional tensor feature 802 as a face feature.

[0063] The reconstruction unit 303 inputs the one-dimensional tensor feature 802 to the MLP 402 and performs calculation processing (linear transformation processing) of the MLP 402 to calculate the one-dimensional tensor feature 803 in which the number of elements in the one-dimensional tensor feature 802 is increased.

[0064] Next, the reconstruction unit 303 inputs the one-dimensional tensor feature 803 to Reshape 403 and performs calculation processing of Reshape 403 to calculate a ``three-dimensional facial feature 804'' that is a three-dimensional facial feature that increases the number of dimensions of the one-dimensional tensor feature 803.

[0065] Next, the reconstruction unit 303 inputs the three-dimensional facial feature amount 804 and the facial image 501 to concatenate 704 and performs the operation of concatenate 704 to generate concatenated information that concatenates the three-dimensional facial feature amount 804 and the facial image 501 as feature amount 805. By concatenate 704, it is possible to reacquire information that was lost during the feature conversion process of ViT 701, thereby improving the accuracy of neural network processing.

[0066] The conversion unit 304 inputs the feature 805 to the ViT 705 and performs arithmetic processing of the ViT 705 to calculate a final output feature 806, which is a one-dimensional tensor feature. The matching unit 203 matches the final output feature 806 obtained by the ViT 705 for one image acquired by the acquisition unit 201 with the final output feature 806 obtained by the ViT 705 for the other image acquired by the acquisition unit 201.

[0067] As in the first embodiment, the above neural network is treated as a single feature transformation model that transforms the features of a face image 501 and outputs a final output feature 806, and the learning method and inference method are also based on the first embodiment.

[0068] In this embodiment, in addition to the effects obtained in the first embodiment, even if information is lost during the feature conversion process, concatenate 704 complements the lost information, thereby improving the accuracy of neural network processing.

[0069] [Third embodiment] In the second embodiment, the reconstruction unit 303 acquires the face image 501 by shortcut combination and connects the output of the conversion unit 302 to the face image 501. However, it is also possible to acquire intermediate features of the conversion unit 302 by shortcut combination and connect the output of the conversion unit 302 to the intermediate features. Furthermore, although the above embodiment shows the case where the present invention is applied to tasks such as face recognition and arbitrary object detection, it may also be applied to other tasks.

[0070] A specific example of a neural network according to this embodiment will be described with reference to FIG. 9. In this embodiment, a case will be described in which an information processing device executes a human pose estimation task, which is a task for estimating the pose of a person included in an image. The input of the human pose estimation task is an image including a person whose pose is to be estimated. The output of the human pose estimation task is the positions of the joints of the person included in the image.

[0071] The conversion unit 302 calculates facial features by inputting a facial image to ResNet401 and performing arithmetic processing of the ResNet401. The reconstruction unit 303 inputs the facial features calculated using ResNet401 to the MLP402 and performs arithmetic processing of the MLP402, thereby converting the facial features.

[0072] In addition, the reconstruction unit 303 inputs the facial feature converted by the MLP 402 to Reshape 903, which performs shape transformation of the matrix (shape transformation to match the shape of the intermediate feature), and performs calculation processing of Reshape 903, thereby converting the facial feature into a facial feature with an increased number of dimensions, similar to Reshape 403.

[0073] The reconstruction unit 303 then inputs the facial features transformed by Reshape 903 and the intermediate features of ResNet 401 to concatenate 704 and performs the operation of concatenate 704. As a result, the reconstruction unit 303 generates concatenated information that concatenates the facial features transformed by Reshape 903 and the intermediate features of ResNet 401 as "optimized features."

[0074] The transform unit 304 transforms the features generated by the concatenate 704 by inputting them to the ViT 705 and performing calculations on the ViT 705. Here, a method for acquiring intermediate features from the ResNet unit 401 will be described with reference to FIG. 10A. First, the ResNet unit 401 is a network that gradually reduces the resolution of an input image, as shown in K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In ECCV, 2016. Each stage shown in FIG. 10A performs feature transformation processing such as convolution, normalization, and pooling, and each downsampling reduces the resolution of the input image by applying a convolution filter with a stride of 2 or more. Here, the intermediate features from Stage 2 are passed to the subsequent concatenate 704 via shortcut connection 1001.

[0075] Furthermore, the feature values input to the reconstruction unit 303 do not necessarily have to be the output from the final layer of the transformation unit 302. For example, as shown in FIG. 10B, intermediate feature values immediately before applying Global Average Pooling (GAP) may be input to the reconstruction unit 303. By using feature values with a large amount of information before compression by GAP, the accuracy of neural network processing can be improved. In this way, the feature values input to the reconstruction unit 303 are not limited to one-dimensional tensors.

[0076] As in the above embodiment, the above neural network is treated as a single feature transformation model that acquires a face image and performs matching using the output from the transformation unit 304 as the final output, and the learning method and inference method are also similar to those in the above embodiment.

[0077] Finally, the effects of this embodiment will be described. In addition to the effects obtained in the above-described embodiment, this embodiment reuses intermediate features of a trained model, thereby preventing the neural network processing from becoming redundant and improving the efficiency of the neural network processing.

[0078] [Fourth embodiment] An example of the configuration of the calculation unit 202 according to this embodiment will be described with reference to the block diagram of Fig. 11. As shown in Fig. 11, the calculation unit 202 according to this embodiment includes a calculation unit 1101 having the same configuration as the calculation unit 202 shown in Fig. 3, a reconstruction unit 1102, and a conversion unit 1103.

[0079] The reconstruction unit 1102 acquires, by shortcut combination, the face image generated by the acquisition unit 301. Then, the reconstruction unit 1102 converts the feature amount output from the conversion unit 1101, and generates, as the feature amount, linked information that links the converted feature amount and the face image generated by the acquisition unit 301.

[0080] The conversion unit 1103 applies feature conversion processing to the feature by inputting the feature generated by the reconstruction unit 1102 to a neural network and performing arithmetic processing on the neural network, thereby converting the feature generated by the conversion unit 1101 into a feature that is less likely to miss features.

[0081] A specific example of a neural network according to this embodiment will be described with reference to Fig. 12. In the first embodiment, matching was performed using features output from ResNet 404, but in this embodiment, feature transformation is further applied to the features. For this further feature transformation, a reconstruction unit 1102 is used.

[0082] The reconstruction unit 1102 converts the facial feature amounts by inputting the feature amounts calculated by the transformation unit 1101 to an MLP1201 similar to the MLP402 and performing arithmetic processing of the MLP1201. The reconstruction unit 1102 then converts the facial feature amounts by inputting the facial feature amounts converted by the MLP1201 to a Reshape1202 similar to the Reshape403 and performing arithmetic processing of the Reshape1202. The reconstruction unit 1102 then inputs the facial feature amounts converted by the Reshape1202 and the facial image generated by the acquisition unit 301 to a concatenate1203 similar to the concatenate704 and performs arithmetic processing of the concatenate1203. As a result, the reconstruction unit 1102 generates concatenated information that concatenates the facial feature amounts converted by the Reshape1202 and the facial image generated by the acquisition unit 301 as an "optimized feature amount."

[0083] The reconstruction unit 1102 acquires, by shortcut combination, the face image generated by the acquisition unit 301. This shortcut combination has the effect of restoring features that are lost as the feature conversion process is repeated, thereby reducing the loss of features in feature extraction.

[0084] The conversion unit 1103 converts the feature amount generated by the concatenate 1203 by inputting the feature amount to the ViT 705 and performing calculation processing of the ViT 705.

[0085] Next, we will explain the inference and training method of the neural network according to this embodiment. The conversion unit 1101 is a neural network obtained by the method of the first embodiment. In the first embodiment, ResNet401 is first trained as an independent neural network. Next, MLP402, Reshape403, and ResNet404 are applied, and then trained as a single neural network. At this time, the weights of ResNet401 are fixed and not updated. In this way, the conversion unit 1101 is obtained.

[0086] Next, the reconstruction unit 1102 and the conversion unit 1103 are applied to the conversion unit 1101, and the result is trained as a single neural network, but at this time, the weights of the conversion unit 1101 are fixed and not updated. In this way, the reconstruction unit 1102 and the conversion unit 1103 are trained.

[0087] During inference, a face image is input to the transform unit 1101, and the output of the transform unit 1103 is set as the final output feature. The matching unit 203 in FIG. 2 performs matching using this final output feature. Note that in this embodiment, an example has been shown in which the training of the ResNet401, transform unit 1101, reconstruction unit 1102, and transform unit 1103 is performed in three stages, but this is not limiting. For example, all may be trained at once. Alternatively, for example, only the ResNet401 may be trained first, and then the transform unit 1101 and the transform unit 1103 may be trained. Alternatively, for example, the transform unit 1103 may be further added, and training may be performed in stages or all at once.

[0088] Finally, we will explain the effects of this embodiment, which are obtained in addition to the effects of the above-mentioned embodiments. In this embodiment, it is possible to gradually learn the output feature values of an existing neural network. Compared to training a large-scale neural network all at once, the number of parameters to be learned at one time can be reduced, which is expected to make learning easier.

[0089] Furthermore, even if there are features that are omitted in the conversion unit 302, the features are complemented by shortcut combination. The conversion unit in the subsequent stage performs feature conversion that combines the omitted features with the high-dimensional features acquired in the previous stage, thereby converting the high-dimensional features acquired in the previous stage into more efficient features. In this way, according to this embodiment, the accuracy of neural network processing can be improved by repeating feature conversion in stages many times.

[0090] [Fifth embodiment] The essence of each of the above embodiments is to efficiently integrate local features and global features by combining local and gradual feature transformation such as CNN with feature transformation with high flexibility in spatial direction such as MLP. There are various possible configurations that satisfy this condition. For example, in the first embodiment, the input image is transformed by CNN and then converted into high-resolution features by MLP. However, this order can also be reversed. Specifically, the input image may first be converted into a feature vector by MLP, and then gradually converted into high-resolution features by multiple deconvolution layers.

[0091] Another possible configuration is a partially modified version of the Pyramid Feature Network, the Stacked Hourglass Network, or the UNet of Ronneberger et al., U-Net: Convolutional Networks for Biomedical Image Segmentation, arXiv:1505.04597. Each of these networks has a symmetrical arrangement of convolutional layers that reduce spatial resolution and deconvolutional layers (or upsampling layers) that increase spatial resolution. In this embodiment, a network is considered in which some or all of these convolutional or deconvolutional layers are replaced with MLP layers. Conceptually, two convolutional layers that reduce spatial resolution are arranged, and the resolution is restored midway through the network. Such a network may achieve the same level of inference performance with less computational effort than before the configuration change. Another possible configuration is one that includes multiple feature transformation layers that do not change spatial resolution between the layers that reduce and increase spatial resolution.

[0092] Furthermore, although the above embodiment has been described with a focus on image recognition, the present invention is not limited to this and may be applied to natural language processing, speech recognition, etc. It can be widely applied to any neural network that processes data having a spatial arrangement relationship or an order relationship.

[0093] In the above embodiment, the conversion unit and reconstruction unit change the spatial size of the image features when processing an image. When applied to an information processing device that processes document data, it is possible to change the feature elements corresponding to the words that make up the document by integrating them. When applied to processing time-series data such as video and audio, it is possible to change the feature elements corresponding to the time direction. Therefore, various information such as images, audio, and documents can be applied as input information to the neural network.

[0094] In this way, an information processing device having a first calculation unit that performs a first transformation, which is a local multi-stage feature transformation, and a second calculation unit that performs a second transformation, which is a feature transformation that is broader than the first transformation (a feature transformation that changes the number of elements and the number of dimensions of a feature in an increasing / decreasing direction different from that of the first transformation), can be applied to various applications.

[0095] The numerical values, processing timing, processing order, processing subject, data (information) configuration / acquisition method / sending destination / sending source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0096] In addition, some or all of the above-described embodiments may be used in appropriate combination, and some or all of the above-described embodiments may be selectively used.

[0097] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0098] The invention of this specification includes the following information processing device, information processing method, and computer program. (Item 1) a first calculation unit that performs a first transformation, which is a local multi-stage feature transformation; a second calculation unit that performs a second transformation that is a feature transformation having a wider range than the first transformation; Equipped with The second transformation is a feature transformation that changes the number of elements and the number of dimensions of the feature in a direction different from that of the first transformation. 1. An information processing device comprising: (Item 2) 2. The information processing device according to item 1, wherein the first calculation unit performs calculation processing in a neural network to which input information is input, and calculates a feature quantity of a one-dimensional tensor. (Item 3) Item 3. The information processing device according to item 2, wherein the second calculation unit performs a linear transformation that increases the number of elements of the feature of the one-dimensional tensor, and calculates a three-dimensional feature by rearranging the elements of the feature obtained by the linear transformation. (Item 4) 4. The information processing device according to item 3, wherein the second calculation unit performs the linear transformation using a fully connected layer. (Item 5) 4. The information processing device according to item 3, wherein the second calculation unit calculates information obtained by fusing the three-dimensional feature amount and the input information as the feature amount. (Item 6) 4. The information processing device according to item 3, wherein the second calculation unit calculates information obtained by combining the two feature amounts in the first calculation unit as the feature amount. (Item 7) 7. The information processing device according to item 6, wherein the two feature amounts are a feature amount of a final layer in the first calculation unit and an intermediate feature amount in the first calculation unit. (Item 8) 7. The information processing device according to item 6, wherein the two feature amounts are two intermediate feature amounts in the first calculation unit. (Item 9) moreover, 9. The information processing device according to any one of items 1 to 8, further comprising a third calculation unit that performs calculation processing in a hierarchical neural network to which the feature values obtained by the second calculation unit are input, and calculates the feature values of a one-dimensional tensor. (Item 10) 10. The information processing device according to item 9, wherein the number of layers in the hierarchical neural network used by the first calculation unit is greater than the number of layers in the hierarchical neural network used by the third calculation unit. (Item 11) moreover, 11. The information processing device according to item 9 or 10, further comprising a matching unit that matches a feature calculated by the third calculation unit for one piece of input information with a feature calculated by the third calculation unit for the other piece of input information. (Item 12) moreover, a fourth calculation unit that converts the feature amount calculated by the third calculation unit and calculates information obtained by fusing the converted feature amount with input information as the feature amount; a fifth calculation unit that performs calculation processing in a hierarchical neural network to which the feature values calculated by the fourth calculation unit are input, and calculates feature values of a one-dimensional tensor; The information processing device according to any one of items 9 to 11, further comprising: (item 13) moreover, Item 13. The information processing device according to item 12, further comprising a comparison unit that compares the feature calculated by the fifth calculation unit for one piece of input information with the feature calculated by the fifth calculation unit for the other piece of input information. (Item 14) 13. The information processing device according to any one of items 5, 6, and 12, wherein the fusion includes concatenation, element multiplication, and addition. (Item 15) moreover, Item 12. The information processing device according to item 11, further comprising a learning means for learning the second calculation unit and the third calculation unit based on the feature amount calculated by the third calculation unit. (Item 16) The information processing device according to any one of items 9, 10, and 12, characterized in that the hierarchical neural network includes ResNet and VisionTransformer. (Item 17) An information processing method performed by an information processing device, a first calculation step in which a first calculation unit of the information processing device performs a first transformation which is a local multi-stage feature transformation; a second calculation step in which a second calculation unit of the information processing device performs a second transformation that is a feature transformation having a wider range than the first transformation; Equipped with The second transformation is a feature transformation that changes the number of elements and the number of dimensions of the feature in a direction different from that of the first transformation. 1. An information processing method comprising: (Item 18) A computer program for causing a computer to function as each means of the information processing device according to any one of items 1 to 16.

[0099] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0100] 301: Acquisition unit 302: Conversion unit 303: Reconstruction unit 304: Conversion unit

Claims

1. a first calculation unit that performs a first transformation, which is a local multi-stage feature transformation; a second calculation unit that performs a second transformation that is a feature transformation having a wider range than the first transformation; Equipped with The second transformation is a feature transformation that changes the number of elements and the number of dimensions of the feature in a direction different from that of the first transformation.

1. An information processing device comprising:

2. The information processing apparatus according to claim 1 , wherein the first calculation unit calculates a feature quantity of a one-dimensional tensor by performing calculation processing in a neural network to which input information is input.

3. 3. The information processing apparatus according to claim 2, wherein the second calculation unit performs a linear transformation that increases the number of elements of the feature quantity of the one-dimensional tensor, and calculates a three-dimensional feature quantity by rearranging the elements of the feature quantity obtained by the linear transformation.

4. The information processing apparatus according to claim 3 , wherein the second calculation unit performs the linear transformation using a fully connected layer.

5. The information processing apparatus according to claim 3 , wherein the second calculation unit calculates information obtained by combining the three-dimensional feature amount and the input information as the feature amount.

6. The information processing apparatus according to claim 3 , wherein the second calculation unit calculates, as the feature, information obtained by combining the two feature amounts in the first calculation unit.

7. 7. The information processing apparatus according to claim 6, wherein the two feature amounts are a feature amount of a final layer in the first calculation unit and an intermediate feature amount in the first calculation unit.

8. The information processing apparatus according to claim 6 , wherein the two feature amounts are two intermediate feature amounts in the first calculation unit.

9. moreover, 2. The information processing apparatus according to claim 1, further comprising a third calculation unit that performs calculation processing in a hierarchical neural network to which the feature values obtained by the second calculation unit are input, to calculate the feature values of a one-dimensional tensor.

10. 10. The information processing apparatus according to claim 9, wherein the number of layers in the hierarchical neural network used by the first calculation unit is greater than the number of layers in the hierarchical neural network used by the third calculation unit.

11. moreover, 10. The information processing apparatus according to claim 9, further comprising a matching unit that matches a feature calculated by the third calculation unit for one piece of input information with a feature calculated by the third calculation unit for the other piece of input information.

12. moreover, a fourth calculation unit that converts the feature amount calculated by the third calculation unit and calculates information obtained by fusing the converted feature amount with input information as the feature amount; a fifth calculation unit that performs calculation processing in a hierarchical neural network to which the feature values calculated by the fourth calculation unit are input, and calculates feature values of a one-dimensional tensor; 10. The information processing apparatus according to claim 9, further comprising:

13. moreover, 13. The information processing apparatus according to claim 12, further comprising a matching unit that matches a feature calculated by the fifth calculation unit for one piece of input information with a feature calculated by the fifth calculation unit for another piece of input information.

14. 6. The information processing apparatus according to claim 5, wherein the fusion includes concatenation, element multiplication, and addition.

15. moreover, 12. The information processing apparatus according to claim 11, further comprising: a learning unit that performs learning of the second calculation unit and the third calculation unit based on the feature amount calculated by the third calculation unit.

16. The information processing apparatus according to claim 9 , wherein the hierarchical neural network includes ResNet and VisionTransformer.

17. An information processing method performed by an information processing device, a first calculation step in which a first calculation unit of the information processing device performs a first transformation which is a local multi-stage feature transformation; a second calculation step in which a second calculation unit of the information processing device performs a second transformation that is a feature transformation having a wider range than the first transformation; Equipped with The second transformation is a feature transformation that changes the number of elements and the number of dimensions of the feature in a direction different from that of the first transformation.

1. An information processing method comprising:

18. A computer program for causing a computer to function as each of the means of the information processing apparatus according to any one of claims 1 to 16.

Images