Information processing device, information processing method, learning model, program, and storage medium

Abstract

An information processing device according to the present disclosure recognizes a target or a state of the target in a captured image, the information processing device comprising: an acquisition means which acquires feature amounts at a plurality of resolutions in the image; a feature amount extraction means which uses a plurality of transformer encoders to extract a noteworthy feature amount based on the feature amounts at the plurality of resolutions; and an output means which outputs, as the recognized result, the target or the state of the target on the basis of the output results of the plurality of transformer encoders. The feature amount extraction means is configured to extract the feature amount of interest among the feature amounts at the plurality of resolutions by inputting, to a transformer encoder associated with one resolution among the plurality of transformer encoders, a first feature amount at one resolution and a second feature amount at another resolution among the feature amounts extracted from the image at the plurality of resolutions.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, a learning model, a program, and a storage medium. [Background technology]

[0002] In recent years, techniques have been proposed that use deep neural networks to recognize the state of objects and people (referred to as targets) within an image (for example, the pose of the target or the direction of the person's gaze).

[0003] Non-Patent Document 1 proposes a technology for recognizing human posture with higher accuracy using a high-resolution network. The high-resolution network exchanges feature information obtained through convolutional processing in parallel high-resolution subnetworks and low-resolution subnetworks. The technology disclosed in Non-Patent Document 1 can recognize human posture with high accuracy by using such a high-resolution network.

[0004] Furthermore, a model (Vision Transformer (ViT)) is known that applies the Transformer model, which exhibits high performance as a module for deep neural networks that process time-series data such as natural language data, to image processing (Non-Patent Literature 2). In Non-Patent Literature 2, the Transformer is applied to image processing by treating the image as sequence data consisting of a series of image patches. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Ke Sun, et al., "Deep High-Resolution Representation Learning for Human Pose Estimation", arXiv:1902.09212v1 [cs.CV], February 25, 2019. [Non-Patent Document 2] "A reliable model for image recognition! A thorough explanation of Vision Transformer (ViT), which has moved beyond CNNs," [online], [Accessed October 19, 2022].<URL: https: / / deepsquare.jp / 2020 / 10 / vision-transformer / #outline__1> [Overview of the project] [Problems that the invention aims to solve]

[0006] The aforementioned non-patent literature describes a method for recognizing targets and their states with relatively high accuracy, but it did not consider a configuration that appropriately utilizes multiple resolution feature quantities in a transformer.

[0007] This invention has been made in view of the above problems, and its object is to provide a technology for recognizing a target or the state of a target with high accuracy. [Means for solving the problem]

[0008] According to the present invention, an information processing device for recognizing a target or the state of a target present in an captured image, An acquisition means for acquiring feature quantities at multiple resolutions of the aforementioned image, A feature extraction means that uses multiple transformer encoders to extract notable features based on the features at the multiple resolutions, Includes output means for outputting the target or the state of the target as a recognition result based on the output results of the plurality of transformer encoders, The feature extraction means is configured to extract a feature quantity of interest among the feature quantities at a plurality of resolutions by inputting a first feature quantity at one resolution among the feature quantities at the plurality of resolutions extracted from the image and a second feature quantity at another resolution among the feature quantities at the plurality of resolutions into a transformer encoder associated with the one resolution among the plurality of transformer encoders. It is possible to provide an information processing apparatus characterized by this.

Advantages of the Invention

[0009] According to the present invention, it becomes possible to provide a technique for accurately recognizing an object or the state of an object. Other features and advantages of the present invention will become apparent from the following description with reference to the accompanying drawings. In the accompanying drawings, the same or similar configurations are denoted by the same reference numerals.

Brief Description of the Drawings

[0010] The accompanying drawings are included in the specification, form a part thereof, show embodiments of the present invention, and are used to explain the principles of the present invention together with the description. [Figure 1] Block diagram showing a functional configuration example of a vehicle according to the present embodiment [Figure 2] Diagram explaining the main configuration for a driving support function in a vehicle according to the present embodiment [Figure 3] Diagram schematically explaining a configuration example of a deep neural network (DNN) model of a model processing unit according to the present embodiment [Figure 4] Diagram schematically explaining a configuration example of a multi-resolution fusion transformer of a DNN model according to the present embodiment [Figure 5] Diagram schematically explaining neural architecture search (NAS) when training a DNN model of a model processing unit according to the present embodiment [Figure 6] Flowchart showing a series of operations of recognition processing in a model processing unit according to the present embodiment [Figure 7]A flowchart illustrating a series of operations in the driver assistance process according to this embodiment. [Modes for carrying out the invention]

[0011] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more of the features described in the embodiments may be combined in any way. Furthermore, identical or similar configurations will be given the same reference numeral, and redundant descriptions will be omitted.

[0012] <Vehicle Configuration> First, an example of the functional configuration of the vehicle 100 according to this embodiment will be described with reference to Figure 1. Note that each of the functional blocks described with reference to the following figures may be integrated or separated, and the functions described may be implemented in other blocks. Furthermore, what is described as hardware may be implemented in software, and vice versa.

[0013] In the following example, the case in which the control unit 108 is incorporated into the vehicle 100 will be explained, but the control unit 108 of the vehicle 100 may be configured as a control module or information processing device having the configuration of the control unit 108. That is, the present invention can be realized as a control module or information processing device having the configuration of a processor 110 and a model processing unit 114 included in the control unit 108.

[0014] The sensor unit 101 includes a camera (imaging means) that outputs images of the area in front of the vehicle 100 (or further to the sides and rear of the vehicle). The sensor unit 101 may further include a Lidar (Light Detection and Ranging) that outputs a distance image obtained by measuring the distance in front of the vehicle (or further to the sides and rear of the vehicle). The sensor unit 101 further includes a camera (imaging means) located inside the vehicle 100 that captures the driver's face. The image of the driver is used, for example, in the inference process of recognizing a target or the state of a target in the model processing unit 114. The sensor unit 101 may also include various sensors that output the acceleration, position information, steering angle, etc., of the vehicle 100.

[0015] The communication unit 102 is a communication device that includes, for example, a communication circuit, and communicates with the information processing server 150 and surrounding traffic systems via mobile communication standardized as, for example, LTE, LTE-Advanced, or so-called 5G. The communication unit 102 obtains trained parameters of the learning model used by the model processing unit 114 from the external information processing server 150. In addition, the communication unit 102 receives some or all of the map data and traffic information from other information processing servers and surrounding traffic systems.

[0016] The control unit 103 includes operating elements such as buttons and touch panels installed inside the vehicle 100, as well as elements that receive input for driving the vehicle 100, such as a steering wheel and brake pedals. The power supply unit 104 includes a battery, such as a lithium-ion battery, and supplies power to various parts of the vehicle 100. The power unit 105 includes, for example, an engine or motor that generates power to drive the vehicle.

[0017] The notification unit 106, when the eye-tracking information processing unit 115 (described later) determines that the driver's condition does not meet predetermined driving standards, notifies the driver of a predetermined sound, such as a warning sound.

[0018] The memory unit 107 includes a non-volatile, high-capacity storage device such as semiconductor memory. It temporarily stores real images output from the sensor unit 101 and other various sensor data output from the sensor unit 101. In addition, the model data acquisition unit 113, described later, stores the trained parameters of the deep neural network (DNN) model executed by the model processing unit 114, which are received, for example, from an external information processing server 150 via the communication unit 102.

[0019] The control unit 108 includes, for example, a processor 110, RAM 111, and ROM 112, and controls the operation of various parts of the vehicle 100. The control unit 108 also acquires images from the sensor unit 101 and performs inference-stage processing, including processing to recognize a target or the state of a target. The control unit 108 enables the functions of various parts of the control unit 108, such as the model processing unit 114, by having the processor 110 load and execute the computer program stored in ROM 112 into RAM 111.

[0020] The processor 110 includes one or more processors, such as a CPU. In addition to the CPU, the processor 110 may include other processors such as a GPU or ASIC for high-speed execution of the processing of the model processing unit 114. The RAM 111 is composed of a volatile storage medium, such as DRAM, and functions as the work memory of the processor 110. The ROM 112 is composed of a non-volatile storage medium and stores computer programs executed by the processor 110 and setting values ​​for operating the control unit 108.

[0021] The model data acquisition unit 113 acquires data of trained parameters for the DNN model from the information processing server 150 and stores it in the storage unit 107. The trained parameters for the DNN model executed by the model processing unit 114 are generated by the DNN model training process in the information processing server 150.

[0022] The model processing unit 114 executes the inference stage processing of the DNN model (which has been trained (optimized) using training data) on the information processing server 150. The DNN model 320 in this embodiment has, for example, the configuration shown in Figure 3.

[0023] The DNN model 320 includes a high-resolution network 311 and a multi-resolution fusion transformer (MRFT) 310, and takes an image 301 as input to output gaze information. The image 301 is, for example, 224 x 224 pixels and consists of, for example, 3-channel RGB data. The gaze information is information that indicates the direction of gaze of a person present in the image 301, as recognized by the DNN.

[0024] The DNN model 320 inputs image 301 into the high-resolution network 311. The high-resolution network 311 applies feature extraction 302, which includes convolution, to image 301. Feature extraction 302, for example, applies batch normalization and ReLU activation to a 3x3 convolution process. Applying feature extraction 302 twice yields a 56x56 size feature (also called a feature map) with 24 channels. The similar rectangular shapes shown in Figure 3 represent feature (feature maps) with their respective sizes and number of channels.

[0025] The high-resolution network 311 then repeats processing using two types of modules. These two modules are the parallel module and the fusion module, and are composed of search blocks, which will be described later. The high-resolution network 311 can obtain larger receptive fields (wider map areas) and features of multiple scales (area sizes) by stacking search blocks in each resolution branch. The parallel module repeatedly extracts features at the highest resolution among multiple resolutions while simultaneously extracting features at lower resolutions among multiple resolutions. The fusion module is placed after the parallel module and exchanges information across multiple resolution branches.

[0026] The search block includes a first search block 303, a second search block 304, and a third search block 305. The first search block includes, for example, a convolution using a 3x3 block, a convolution using a 5x5 block, and a convolution using a 7x7 block. The second search block includes, for example, a convolution using a 3x3 block and a convolution using a 5x5 block. The third search block includes, for example, a convolution using a 3x3 block.

[0027] In the high-resolution network 311, the lowest resolution feature branch (e.g., 14x14) is generated from a branch with a lower resolution feature (e.g., 28x28) that is one level higher in resolution. Branches of adjacent resolutions are connected to each other via search blocks, and the features of each branch are merged. For example, the feature output from the second branch (28x28) incorporates the feature input from the first branch (56x56), the feature input from the second branch (28x28), and the feature input from the third branch (14x14).

[0028] The high-resolution network 311 of the DNN model 320 gradually adds branches of lower-resolution features using parallel modules and fusion modules, and fuses the information from branches of multiple resolutions.

[0029] The high-resolution network 311 reduces the dimensionality of the feature channels by applying a 1x1 convolutional layer 306 that performs point-by-point convolution in each resolution branch. Reducing the dimensionality of the feature channels reduces the computational complexity in subsequent recognition processing. The high-resolution network 311 outputs feature quantities (feature maps) 307, 308, and 309 corresponding to each resolution branch. The DNN model 320 inputs the feature quantities 307, 308, and 309 from the high-resolution network 311 into the multi-resolution fusion transformer 310.

[0030] The multi-resolution fusion transformer 310 extracts notable feature amounts from each of the feature amounts at multiple resolutions and outputs gaze information 312 including the gaze direction of a person as a recognition result. Referring to FIG. 4, the configuration of the multi-resolution fusion transformer (MRFT) 310 according to the present embodiment will be described. MRFT 310 is included in the DNN model 320 configured in the model processing unit 114.

[0031] The feature amounts 307, 308, and 309 at three resolutions input to MRFT 310 are the outputs of the high-resolution network 311 as described above. MRFT 310 changes the size of the feature amounts output from the high-resolution network 311, aggregates them, and connects them to the transformer encoder. The transformer encoder uses the self-attention mechanism to obtain the correlation between patches. The transformer encoder can model the feature amounts at multiple resolutions to some extent even by simply concatenating them. However, the strong correlation between the feature amounts at different resolutions is not sufficiently extracted by the original transformer. For this reason, the present embodiment has the configuration shown in FIG. 4.

[0032] MRFT 310 reforms the feature amounts at all resolutions into a sequence of flattened two-dimensional patches by each of PE410 to 430. Here, the feature amounts 307, 308, and 309 are h i ×w i ×c i in dimension. h i ×w i indicates the resolution of the i-th feature amount, and c i indicates the number of channels of the i-th feature amount. The sequence of two-dimensional patches generated by PE410 to 430 is n i ×(p i 2 ·c i ) in dimension, and p i ×p i indicates the resolution of the feature amount patch. n i is the number of generated feature amount patches, and n i =h i w i / pi 2 This sequence of patches also functions as the effective input sequence length for the transformer encoder.

[0033] The MRFT310 inputs the generated feature patch sequences into transformer encoders 411, 421, and 431, respectively. Each of the transformer encoders 411, 421, and 431 includes MHSA412, Add&Normalization413, FFN414, and Add&Normalization415.

[0034] MRFT310 performs a linear transformation on each sequence of flattened 2D patches to obtain a feature query q. i , the key k of the feature i , and value v i It maps to three matrices. The transformer queries are generated using concatenation 416, 426, and 436 such that Q1=T1(q2++q3), Q2=T2(q1++q3), and Q3=T3(q1++q2). Here, ++ is a per-channel concatenation operator, and T i This represents the conversion function. In this case, the MRFT310 takes the input as key k i This converts them to the same size. By performing this concatenation, the low-resolution features are enhanced by other high-resolution features, which mainly consist of global features, and the high-resolution features are provided with local information from the other low-resolution features.

[0035] The MRFT310 takes a feature vector at a certain resolution (first feature vector) as the key and value of the transformer encoder, concatenates features from other resolutions among multiple resolutions (second feature vector), and inputs this as the query for the transformer encoder. This allows the MRFT310 to extract the first feature vector that is highly correlated with the second feature vector using the modeled correlation. By sharing features vectors at different resolutions, it can efficiently output results when there is a strong correlation between features vectors at different resolutions.

[0036] The operations of transformer encoders 411, 421, and 431 are expressed by Equation 1.

number

number

[0037] The MRFT310 has output X i out By applying a Global Average Pooling (GAP) 440 layer and an MLP (Multilayer Perceptron) 441 layer, the final gaze information 312 is output. The GAP 440 layer outputs X i out The values ​​are added together with adjusted resolution, and the average value is calculated. This allows for the identification of specific output values. The MLP441 receives the output results of multiple transformer encoders via the GAP440. The MLP441 includes multiple neural network layers and is trained to output gaze information 312 based on these output results.

[0038] The line-of-sight information 312 includes, for example, xy coordinate values or xy direction angles when the center of the rectangle of the face in the image or the intermediate position between the left and right eyes is used as the origin, assuming that the angle of the line of sight looking at the camera that captures the face in the image is 0 degrees and there is no inclination.

[0039] Referring to FIG. 1 again for explanation. The line-of-sight information processing unit 115 executes a driving support function based on the line-of-sight information 312 output from the MRFT 310. The driving support function includes, for example, a warning against sidelong glances by the driver. It determines whether the position or movement of the person's line of sight meets a predetermined driving standard, and generates a notification when the predetermined driving standard is not met. This example is an example of a driving support function that uses the line-of-sight information output from the MRFT 310, and the driving support function may be other functions as long as they use the line-of-sight information. The driving support function in the present embodiment can be realized using known techniques as long as it uses the line-of-sight information. An example of the driving support function by the line-of-sight information processing unit 115 will be described later.

[0040] <Process of training the MRFT> Referring to FIG. 5, the processing in the learning stage of the DNN model 320 will be described. In the present embodiment, the processing in the learning stage of the DNN model 320 will be described by taking the case where it is executed in the information processing server 150 as an example. However, in the vehicle 100, the control unit 108 may execute the processing in the learning stage of the DNN model 320.

[0041] For the training of the DNN model 320, for example, NAS (Neural Architecture Search) is used to search for and optimize the parameters of the architecture including not only the weight parameters of the normal neural network but also the hyperparameters of the DNN model. In the present embodiment, the case where NAS is used for the training of the DNN model 320 is described as an example, but a method of training the DNN model 320 after determining the architecture and hyperparameters of the DNN in advance may also be used.

[0042] The exploration block in the NAS includes three paths: MixConv502, residual connection path 530, and lightweight transformer 503. The lightweight transformer extracts the global context. In this embodiment, the number of convolutional channels in MixConv502 and the number of tokens in the lightweight transformer are exploreable parameters.

[0043] In this embodiment, MixConv502 provides 3x3, 5x5, and 7x7 kernel search blocks. A channel of depth-direction convolution or a token of a lightweight transformer is sometimes referred to as a search unit. In the example in Figure 5, the input c501 of the search block is a channel of c features. To enhance the feature representation of input c501, an SE (Squeeze-and-Excitation) block 504 is applied. In the MixConv502 pass, the input channel is expanded to (r3+r5+r7)c by a point-by-point 1x1 convolution. i indicates the scaling factor of the i×i convolution. The output is r i The kernel is divided accordingly and fed into convolutions 511-513 with depths of 3x3, 5x5, and 7x7 respectively. When convolution is performed by convolutions 511-513, the outputs from all convolutions 511-513 are concatenated. Next, another 1x1 convolution is applied to the concatenated result to shrink the channel to match the desired output channel c'.

[0044] In the path of the lightweight transformer 503, the projector 521 is used to project the input features of size c×h×w to a reduced size n×s×s, thereby converting them to the size to be input to the transformer. The projector 521 is used to reduce computational cost, where n represents the number of queries and s×s represents the reduced spatial size. The inverse projector 524 is applied to the outputs of the transformer's encoder 522 and decoder 523 to backproject the outputs to the desired output size.

[0045] Furthermore, in this embodiment, a residual connection path 530 is provided in the search block. The residual connection path 530 can handle the case where all search units in the search block become zero during the search. The residual connection path 530 applies a point-by-point 1x1 convolution to obtain the desired output size. The outputs of MixConv 502, lightweight transformer 503, and residual connection path 530 are concatenated and output.

[0046] When running NAS using the configuration shown in Figure 5, in this embodiment, for example, known progressive shrinking can be used. With progressive shrinking, the entire network is first trained, and then the configuration, such as the number of channels, can be fine-tuned. In this embodiment, progressive shrinking can reduce the number of convolutional channels and the number of transformer queries through the training phase, thereby generating a lightweight DNN model. More specifically, training is performed using a loss function as follows, with a penalty value weighted by the amount of computational cost reduced during training.

number

[0047] As mentioned above, the DNN model 320 can be used for real-time gaze recognition because it has a small number of transformer encoders and the computational load is reduced through optimization by NAS.

[0048] <Main components for driver assistance functions in vehicles> Next, with reference to Figure 2, the main configuration for the driver assistance function in vehicle 100 will be described. The model data acquisition unit 113 of vehicle 100 acquires the trained parameters of the DNN model described above (weight parameters and architecture parameters optimized through training) from, for example, the information processing server 150. The acquired trained parameters are used by the model processing unit 114.

[0049] The sensor unit 101 captures and outputs a photograph of the driver's face. The model processing unit 114 of the control unit 108 uses the image to perform the gaze recognition process described above and outputs gaze information. The sensor unit 101 also captures and outputs front and side images of the vehicle 100, and the control unit 108, for example, recognizes objects in the image and their three-dimensional positions relative to the camera. The gaze information processing unit 115 uses the three-dimensional position of the object and the gaze information to identify the object that the driver is looking at. The gaze information processing unit 115 determines that the predetermined driving standards are not met if the object the driver is looking at is a predetermined object that does not meet the predetermined driving standards, or if it is outside the range of gaze directions necessary for normal driving. The notification unit 106, in response to the determination of the gaze information processing unit 115, for example, notifies the driver with a warning sound.

[0050] <Operation of gaze recognition processing in the model processing unit> Referring to Figure 6, a series of operations for gaze recognition processing in the model processing unit 114 will be described. Note that gaze recognition processing is implemented, for example, by the processor 110 loading a computer program stored in the ROM 112 or memory unit 107 into the RAM 111 and executing it. Unless otherwise specified, the model processing unit 114 operates as the primary processor in the following operations.

[0051] In S601, the model processing unit 114 acquires the driver's face image captured by the sensor unit 101. In S602, the model processing unit 114 extracts feature quantities of multiple resolutions by processing using the high-resolution network 311 described above. The model processing unit 114 outputs the feature quantities 307, 308, and 309 of the multiple resolutions described above.

[0052] In S603, the model processing unit 114 processes the extracted feature quantities at multiple resolutions using multiple transformer encoders 411, 421, and 431. As a result, the model processing unit 114 generates multiple output X values ​​at multiple resolutions, which have been processed by the transformer encoders. i out Outputs (the output taking into account the correlation between features).

[0053] In S604, the model processing unit 114 applies average pooling to the output of multiple resolutions using the GAP440. In S605, the MLP441 outputs the direction of the person's gaze as the recognition result. Once the model processing unit 114 outputs the recognition result, it terminates the gaze recognition process.

[0054] <Operation of driver assistance processing in vehicles> Next, with reference to Figure 7, the operation of the driver assistance processing in the vehicle 100 will be described. The driver assistance processing is implemented, for example, by the processor 110 loading a computer program stored in the ROM 112 or memory unit 107 into the RAM 111 and executing it.

[0055] In S701, the sensor unit 101 acquires an image of a person's face inside the vehicle. In S702, the model processing unit 114 performs the gaze recognition process described above to recognize the direction of the person's gaze in the image.

[0056] In S703, the gaze information processing unit 115 determines whether the driver's gaze or gaze movement meets predetermined driving standards based on the gaze direction of the person recognized by the model processing unit 114. The gaze information processing unit 115 may further use images taken from the front or side of the vehicle 100, the recognition results of objects based on said images, distance information from the camera to the recognized object, etc. As described above, the gaze information processing unit 115 determines that the predetermined driving standards are not met if the object the driver is looking at is a predetermined object that does not meet the predetermined driving standards, or if it is outside the range of gaze directions necessary for normal driving.

[0057] In S704, the eye-gaze information processing unit 115 determines that the driver's gaze or eye movements meet predetermined driving criteria, and returns to S701; otherwise, it proceeds to S705. In S705, the notification unit 106, in accordance with the determination of the eye-gaze information processing unit 115, notifies the driver, for example, of a warning sound. In this way, the eye-gaze information obtained by the model processing unit 114 can be used to assist the driver. The eye-gaze information processing unit 115 then terminates the driving assistance process.

[0058] In the above-described embodiment, the configuration of the DNN model 320 was used to explain the case of recognizing the gaze direction of a person in an image. However, the configuration of the DNN model 320 can be used to recognize targets or the state of targets in an image, not limited to the gaze direction of a person.

[0059] Furthermore, in the above-described embodiment, the case in which the model processing unit 114 inside the vehicle executes the processing of the DNN model 320 was explained as an example. However, the processing of the DNN model 320 is not limited to execution in the vehicle 100, but can also be executed by an external information processing server. In this case, for example, the vehicle 100 may transmit an image of the driver to the information processing server, and the information processing server may execute the processing of the DNN model 320 to recognize the direction of gaze. In other words, the processing of the DNN model 320 according to this embodiment may be executed by the control unit 108 as an information processing device, or by an information processing device different from the control unit 108 mounted in the vehicle 100. Also, the processing of the DNN model 320 according to this embodiment may be executed by an information processing server as an information processing device.

[0060] Furthermore, while the above-mentioned DNN model 320 was explained using the example of extracting features of three different resolutions, other numbers of resolutions, such as four, may also be used. However, since the computational complexity increases with the number of resolution types, the number of resolution types should be four or less.

[0061] As described above, in this embodiment, the information processing device recognizes a target or the state of a target present in the captured image. The information processing device extracts feature quantities at multiple resolutions of the image and uses multiple transformer encoders to extract noteworthy feature quantities based on the feature quantities at multiple resolutions. The information processing device further outputs the target or the state of a target as a recognition result based on the output results of the multiple transformer encoders. At this time, the first feature quantity at one resolution and the second feature quantity at another resolution among the feature quantities at multiple resolutions extracted from the image are input to a transformer encoder associated with one of the multiple transformer encoders, thereby extracting noteworthy feature quantities among the feature quantities at multiple resolutions. In this way, the target or the state of a target can be recognized with high accuracy.

[0062] The invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of the gist of the invention. This application claims priority based on Japanese Patent Application No. 2022-205965, filed on December 22, 2022, and all of its contents are incorporated herein by reference. [Explanation of symbols]

[0063] 100...Vehicle, 101...Sensor unit, 108...Control unit, 113...Model data acquisition unit, 114...Model acquisition unit

Claims

1. An information processing device that recognizes a target or the state of a target present in an captured image, An acquisition means for acquiring feature quantities at multiple resolutions of the aforementioned image, A feature extraction means that uses multiple transformer encoders to extract notable features based on the features at the multiple resolutions, Includes output means for outputting the target or the state of the target as a recognition result based on the output results of the plurality of transformer encoders, The information processing apparatus is characterized in that the feature extraction means is configured to extract a noteworthy feature from the feature at multiple resolutions by inputting a first feature at one of the multiple resolutions extracted from the image and a second feature at another resolution among the multiple resolutions into a transformer encoder associated with one of the multiple transformer encoders.

2. The information processing apparatus according to claim 1, characterized in that the feature extraction means inputs the first feature as key and value for the transformer encoder, inputs the second feature as a query for the transformer encoder, and extracts the first feature that is highly correlated with the second feature.

3. The information processing apparatus according to claim 1, characterized in that the feature extraction means inputs a feature obtained by concatenating the feature quantities at other resolutions among the feature quantities at the multiple resolutions to a transformer encoder associated with one of the resolutions as a feature quantity at the other resolutions.

4. The information processing apparatus according to claim 1, characterized in that each of the multiple transformer encoders is associated with each of the different resolutions of the multiple resolutions.

5. The information processing apparatus according to claim 1, characterized in that the number of transformer encoders corresponds to the number of resolution types in the plurality of resolutions.

6. The information processing apparatus according to claim 1, characterized in that the number of transformer encoders is four or less.

7. The information processing apparatus according to claim 1, characterized in that the plurality of transformer encoders are not connected in series with one another.

8. The information processing apparatus according to claim 1, characterized in that the output means includes a network layer which is learned to output the target or the state of the target as a recognition result based on the output results of the plurality of transformer encoders.

9. The information processing apparatus according to claim 8, characterized in that the output means inputs to the network layer the result of applying a pooling process using an average value to the output results from each of the plurality of transformer encoders.

10. The information processing apparatus according to claim 1, characterized in that the target includes a person's face, and the state of the target includes the direction of gaze at the person's face.

11. The information processing apparatus according to claim 1, wherein the acquisition means includes a second feature extraction means that extracts feature quantities at multiple resolutions of the image using a neural network.

12. The information processing apparatus according to claim 11, wherein the second feature extraction means uses a high-resolution network that repeatedly extracts features at the highest resolution among the plurality of resolutions, while simultaneously extracting features at a lower resolution among the plurality of resolutions, and exchanges features between the resolutions.

13. An information processing method performed in an information processing device for recognizing a target or the state of a target present in an captured image, An acquisition step to acquire feature quantities at multiple resolutions of the aforementioned image, A feature extraction step that uses multiple transformer encoders to extract notable features based on the features at the multiple resolutions, The output step includes outputting the target or the state of the target as a recognition result based on the output results of the plurality of transformer encoders, An information processing method configured to extract a noteworthy feature from the multiple resolutions by inputting a first feature at one of the multiple resolutions extracted from the image, and a second feature at another resolution, to a transformer encoder associated with one of the multiple transformer encoders.

14. A learning model configured to recognize an object or the state of an object present in an image, A first neural network including multiple transformer encoders that take feature quantities at multiple resolutions of the aforementioned image as input and extract notable feature quantities based on the feature quantities at multiple resolutions, The system includes a second neural network that is trained to output the target or the state of the target as a recognition result based on the output results of the plurality of transformer encoders, A learning model characterized in that the plurality of transformer encoders cause a computer to function to extract a noteworthy feature from the plurality of resolutions by inputting a first feature at one of the plurality of resolutions extracted from the image and a second feature obtained by concatenating a first feature at one of the plurality of resolutions and a second feature obtained by concatenating a feature at another of the plurality of resolutions into a transformer encoder associated with one of the plurality of transformer encoders.

15. A program for causing a computer to function as one of the means of an information processing apparatus according to any one of claims 1 to 12.

16. A storage medium for storing a program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 12.

Images