Information processing device, information processing method, and program

Abstract

It accurately determines the correspondence between images and text. [Solution] In the information processing device, the visual feature extraction means extracts visual features from visual information. The text feature extraction means extracts text features from text corresponding to the visual features. The feature adjustment means performs feature alignment of text features and visual features based on the interrelationship between text features and visual features.

Description

Technical Field

[0001] The present disclosure relates to a technique for associating visual information with text.

Background Art

[0002] There is known a technique for extracting image features from an image, extracting text features from text, and comparing them to determine the correspondence between the image and the text. For example, Patent Document 1 describes a method of learning a model so as to embed a sentence indicating the content of an image and the image in a common space and performing an image search using this model.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In order to accurately determine the correspondence and similarity between an image and text, it is necessary to accurately embed the image features and the text features in the same feature space.

[0005] One object of the present disclosure is to provide an information processing apparatus capable of accurately determining the correspondence between an image and text.

Means for Solving the Problems

[0006] In one aspect of the present disclosure, the information processing apparatus includes visual feature extraction means for extracting visual features from visual information, text feature extraction means for extracting text features from text corresponding to the visual features, feature adjustment means for performing feature alignment of the text features and the visual features based on the mutual relationship between the text features and the visual features, It is equipped with.

[0007] From another perspective of this disclosure, computer-based information processing methods are: Extract visual features from visual information, Text features are extracted from the text corresponding to the aforementioned visual features. Based on the interrelationship between the text features and the visual features, feature alignment is performed on the text features and the visual features.

[0008] In yet another aspect of this disclosure, the program is Extract visual features from visual information, Text features are extracted from the text corresponding to the aforementioned visual features. Based on the interrelationship between the text features and the visual features, the computer is instructed to perform a feature alignment process for the text features and the visual features. [Effects of the Invention]

[0009] According to this disclosure, it becomes possible to accurately associate images with text. [Brief explanation of the drawing]

[0010] [Figure 1] The overall configuration of an information processing device relating to an example of this disclosure is shown. [Figure 2] This is a block diagram showing the hardware configuration of an information processing device. [Figure 3] This is a block diagram showing the functional configuration of an information processing device. [Figure 4] An example of classification by the classification unit is shown. [Figure 5] This shows block diagrams of variations of the feature adjustment block. [Figure 6] This shows block diagrams of variations of the feature adjustment block. [Figure 7] An example of the configuration of the mutual feature adjustment unit is shown. [Figure 8] This is a block diagram showing a specific example of the mutual feature adjustment unit. [Figure 9] Shows a flowchart of the processing by the mutual feature adjustment unit. [Figure 10] Shows variations of the mutual feature adjustment unit. [Figure 11] Shows variations of the mutual feature adjustment unit. [Figure 12] Shows the configuration during learning of the information processing apparatus. [Figure 13] It is a flowchart of the classification processing. [Figure 14] Shows the results of 5-class classification by the information processing apparatus. [Figure 15] Shows the distribution of features in the feature space for the comparative example and the proposed method. [Figure 16] Shows an example of a behavior management system to which the information processing apparatus is applied. [Figure 17] It is a block diagram showing the functional configuration of another information processing apparatus. [Figure 18] It is a flowchart of the processing by another information processing apparatus.

Mode for Carrying Out the Invention

[0011] <First Embodiment> [Overall Configuration] FIG. 1 shows the overall configuration of an information processing apparatus according to an embodiment of the present disclosure. The information processing apparatus 100 associates input visual information and text. Specifically, the information processing apparatus 100 determines the text corresponding to the input visual information among a plurality of texts included in the input text group. The visual information may be an image (still image) or a video (moving image).

[0012] Visual information captured by a camera or the like is input to the information processing apparatus 100. For the information processing apparatus 100, the visual information may be directly input from the camera, or the visual information stored in a database or the like may be input. Further, a text group corresponding to the content of the visual information is input to the information processing apparatus 100.

[0013] In one example, the visual information is information captured of a person performing some action, and the text consists of the name of the action and a description of that action. In this case, the information processing device 100 outputs the text describing the person's action included in the visual information as the classification result.

[0014] [Hardware configuration] Figure 2 is a block diagram showing the hardware configuration of the information processing device 100. As shown in the figure, the information processing device 100 comprises a processor 11, an interface (IF) 12, a ROM (Read Only Memory) 13, a RAM (Random Access Memory) 14, a database (DB) 15, and a recording medium 16. Each component is connected to the others, for example, via a bus 18.

[0015] The processor 11 is a computer such as a CPU (Central Processing Unit) and controls the entire information processing device 100 by executing a pre-prepared program. Specifically, the processor 11 can be a CPU, GPU (Graphics Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof.

[0016] Furthermore, the processor 11 loads programs stored in the ROM 13 and recording medium 16 into the RAM 14 and executes each process coded in the program. The processor 11 functions as part or all of the information processing device 100. Specifically, the processor 11 performs the classification process described later.

[0017] IF12 transmits and receives data to and from external devices. Specifically, the information processing device 100 receives text data and visual information through IF12. The information processing device 100 also outputs the classification results of the visual information to a display device or other external devices through IF12.

[0018] ROM 13 stores various programs executed by processor 11. RAM 14 is used as working memory while processor 11 is executing various processes.

[0019] DB15 stores various algorithms, data, machine learning models, etc., that the information processing device 100 uses when performing the classification process described later.

[0020] The recording medium 16 is a non-volatile, non-temporary recording medium such as a disk-shaped recording medium or semiconductor memory. The recording medium 16 may be configured to be detachable from the information processing device 100. The recording medium 16 stores various programs executed by the processor 11.

[0021] In addition to the above, the information processing device 100 may also be equipped with a display device such as a liquid crystal display, and an input device such as a keyboard or mouse. These display devices and input devices are used, for example, by the operator of the information processing device 100.

[0022] [Functional Configuration] (Basic configuration) Figure 3 is a block diagram showing the functional configuration of the information processing device 100. As shown in the figure, the information processing device 100 includes a text feature extraction unit 21, a visual feature extraction unit 22, a text feature adjustment unit 23, a visual feature adjustment unit 24, a mutual feature adjustment unit 25, and a classification unit 26.

[0023] The text feature extraction unit 21 receives a text group TX as input. The text group TX includes multiple labels corresponding to visual information. For example, when the information processing device 100 classifies human work, the text group TX includes labels indicating human work, such as compaction, trolley transport, etc. The text feature extraction unit 21 extracts text features TF from each of the multiple texts included in the text group TX and outputs them as a text feature group TF to the text feature adjustment unit 23. Text features are feature quantities extracted from text data, and include vectors that represent text numerically. For example, text features include Bag of Words (BoW), TF-IDF (Term Frequency-Inverse Document Frequency), and word embeddings. In this embodiment, the text feature extraction unit 21 uses a pre-trained base model, and the text feature extraction unit 21 is not subject to training.

[0024] The visual feature extraction unit 22 extracts visual features VF from the visual information VI. The visual information VI includes images (still images) or videos (video). The visual feature extraction unit 22 extracts visual features from the frame images that make up the visual information VI and outputs them as visual features VF to the visual feature adjustment unit 24. Visual features are feature quantities that represent images or videos in numerical form. Examples of visual features include feature vectors generated by CNN (Combolutional Neural Network), VGG, ResNet, etc. In this embodiment, the visual feature extraction unit 22 uses a pre-trained base model and is not subject to training.

[0025] The text feature adjustment unit 23 obtains the text feature group TF from the text feature extraction unit 21 and adjusts the text feature group TF. Specifically, the text feature adjustment unit 23 embeds the text feature group TF into the feature space. At that time, the text feature adjustment unit 23 performs feature alignment in the feature space. "Feature alignment" refers to aligning features of different distributions or formats into the same feature space. Specifically, the text feature adjustment unit 23 adjusts the position of each text feature in the feature space. The text feature adjustment unit 23 performs feature alignment on each text feature and outputs the text feature group TFa after feature alignment to the mutual feature adjustment unit 25 and the classification unit 26.

[0026] The visual feature adjustment unit 24 acquires the visual feature VF from the visual feature extraction unit 22 and adjusts the visual feature VF. Specifically, the visual feature adjustment unit 24 embeds the visual feature VF into the same feature space as the text feature TF. That is, the text feature TF and the visual feature VF are embedded in a common feature space. At that time, the visual feature adjustment unit 24 adjusts the position of the visual feature VF in the feature space. That is, the visual feature adjustment unit 24 performs feature alignment of the visual features and outputs the visual feature VFa after feature alignment to the mutual feature adjustment unit 25 and the classification unit 26.

[0027] The mutual feature adjustment unit 25 obtains the text feature group TF from the text feature adjustment unit 23 and the visual feature VF from the visual feature adjustment unit 24. Then, the mutual feature adjustment unit 25 performs feature alignment between the text feature group TF and the visual feature VF based on their interrelationships. "Interrelationship" is a concept that shows how features are related to each other, and specifically refers to dependency relationships, correlations, common patterns between features, and the presence or absence of such patterns. The mutual feature adjustment unit 25 generates feature alignment information that shows the interrelationships between the text feature group TF and the visual feature VF, and performs feature alignment between the text feature group TF and the visual feature VF by applying the generated feature alignment information to the text feature group TF and the visual feature VF. The feature alignment performed by the mutual feature adjustment unit 25 is also called "mutual feature alignment." That is, "mutual feature alignment" refers to feature alignment between text features and visual features. The mutual feature adjustment unit 25 outputs the text feature group TF after mutual feature alignment to the text feature adjustment unit 23, and outputs the visual feature VF after mutual feature alignment to the visual feature adjustment unit 24.

[0028] The mutual feature adjustment unit 25 can be configured by inputting the input or output of the text feature adjustment unit 23 and the visual feature adjustment unit 24, or the intermediate features of the text feature adjustment unit 23 and the visual feature adjustment unit 24. That is, in the first example, as shown by arrow 27a in Figure 3, the input to the text feature adjustment unit 23 and the input to the visual feature adjustment unit 24 are input to the mutual feature adjustment unit 25. In the second example, as shown by arrow 27b in Figure 3, the output of the text feature adjustment unit 23 and the output of the visual feature adjustment unit 24 are input to the mutual feature adjustment unit 25. In the third example, as shown by arrow 27c in Figure 3, the intermediate features of the text feature adjustment unit 23 and the intermediate features of the visual feature adjustment unit 24 are input to the mutual feature adjustment unit 25. Note that "intermediate features" refer to features obtained within a machine learning model or deep learning model, and specifically, are feature quantities extracted during the process of data passing through multiple layers within those models. Here, the output from either of the intermediate layers in the model constituting the text feature adjustment unit 23 and the visual feature adjustment unit 24 is used as an intermediate feature.

[0029] The text feature adjustment unit 23 outputs the text feature group TFa after feature alignment to the classification unit 26. The visual feature adjustment unit 24 also outputs the visual features VFa after feature alignment to the classification unit 26.

[0030] The classification unit 26 classifies visual information using the text feature group TFa input from the text feature adjustment unit 23 and the visual features VFa input from the visual feature adjustment unit 24. Specifically, the classification unit 26 classifies the visual information into one of several texts based on the similarity between multiple text features included in the text feature group TFa and the visual features, i.e., the distance in the feature space.

[0031] Figure 4 shows an example of classification by the classification unit 26. Let's assume that an image or video of a person working at a work site is input as visual information. Let's also assume that five texts describing human work, namely "compaction," "cart transport," "frame assembly," "ground leveling work," and "heavy machine excavation," are input as a set of texts. In this case, the classification unit 26 calculates the similarity between the visual features and the text features of the five texts, and determines that the text with the highest similarity (in the example of Figure 4, "ground leveling work") corresponds to the visual information.

[0032] (Variations of the feature adjustment block) The above-mentioned text feature adjustment unit 23, visual feature adjustment unit 24, and mutual feature adjustment unit 25 (hereinafter, the three are collectively referred to as the "feature adjustment block") can be configured as follows.

[0033] Figure 5(A) shows a block diagram of the first variation of the feature adjustment block. In the first variation, a mutual feature adjustment unit 25 is placed after the text feature adjustment unit 23a and the visual feature adjustment unit 24a, and then a text feature adjustment unit 23b and a visual feature adjustment unit 24b are placed after that. As a result, text features are adjusted in three stages: text feature adjustment unit 23a, mutual feature adjustment unit 25, and text feature adjustment unit 23b. Similarly, visual features are adjusted in three stages: visual feature adjustment unit 24a, mutual feature adjustment unit 25, and visual feature adjustment unit 24b. Note that the text feature adjustment units 23a and 23b have the same configuration but are networks with different parameters. Similarly, the visual feature adjustment units 24a and 24b have the same configuration but are networks with different parameters.

[0034] Figure 5(B) shows a block diagram of the second variation of the feature adjustment block. In the second variation, a text feature adjustment unit 23 and a visual feature adjustment unit 24 are placed between the two mutual feature adjustment units 25a and 25b. As a result, text features are adjusted in three stages: mutual feature adjustment unit 25a, text feature adjustment unit 23, and mutual feature adjustment unit 25b. Similarly, visual features are adjusted in three stages: mutual feature adjustment unit 25a, visual feature adjustment unit 24, and mutual feature adjustment unit 25b. Note that mutual feature adjustment units 25a and 25b have the same configuration but are networks with different parameters.

[0035] Figure 6(A) shows a block diagram of the third variation of the feature adjustment block. In the third variation, the mutual feature adjustment unit 25 adjusts visual features but not text features. That is, text features are adjusted in two stages by the text feature adjustment unit 23a and the text feature adjustment unit 23b. On the other hand, visual features are adjusted in three stages by the visual feature adjustment unit 24a, the mutual feature adjustment unit 25, and the visual feature adjustment unit 24b. Note that the text feature adjustment units 23a and 23b have the same configuration but are networks with different parameters. Similarly, the visual feature adjustment units 24a and 24b have the same configuration but are networks with different parameters.

[0036] Figure 6(B) shows a block diagram of the fourth variation of the feature adjustment block. In the fourth variation, no adjustment is made to text features. On the other hand, visual features are adjusted in three stages: the mutual feature adjustment unit 25a, the visual feature adjustment unit 24, and the mutual feature adjustment unit 25b. Although the mutual feature adjustment units 25a and 25b have the same configuration, they are networks with different parameters.

[0037] (Inter-characteristic adjustment unit) Next, the mutual feature adjustment unit 25 will be described in detail. The mutual feature adjustment unit 25 generates feature alignment information based on the interrelationship between text features and visual features. Figure 7 shows an example of the mutual feature adjustment unit 25. Note that Figure 7 shows a configuration in which the mutual feature adjustment unit 25 is sandwiched between two pairs, the text feature adjustment unit 23 and the visual feature adjustment unit 24, as shown in Figure 6(A), but the feature adjustment unit may have other configurations.

[0038] As shown in the figure, the mutual feature adjustment unit 25 comprises mutual attention mechanisms 31 and 32 and conversion units 33 and 34. The mutual attention mechanisms 31 and 32 calculate the interrelationship between the text feature group and the visual features and output it as feature alignment information AL to the conversion units 33 and 34, respectively. The conversion units 33 and 34 perform predetermined conversions on the input feature alignment information AL. Furthermore, the mutual feature adjustment unit 25 synthesizes the feature alignment information AL converted by the conversion unit 33 with the text feature group and outputs the text features TFa after feature alignment. In addition, the mutual feature adjustment unit 25 synthesizes the feature alignment information AL converted by the conversion unit 34 with the visual features and outputs the visual features VFa after feature alignment.

[0039] Figure 8 is a block diagram showing a specific example of the mutual feature adjustment unit 25. As shown in the figure, the mutual feature adjustment unit 25 comprises a mutual feature adjustment unit 25c that performs feature alignment of text features and a mutual feature adjustment unit 25d that performs feature alignment of visual features. For the sake of explanation, Figure 8 shows the internal configuration of the mutual feature adjustment unit 25d. The mutual feature adjustment unit 25d comprises a mutual attention mechanism 32 and a conversion unit 34. Visual features are input to the mutual attention mechanism 32 as query q, and groups of text features are input as key k and value v. The mutual attention mechanism 32 extracts visual features that are highly related to the groups of text features and outputs them to the conversion unit 34. The conversion unit 34 is composed of, for example, a linear function or an activation function, and converts the text features output by the mutual attention mechanism 32 into weights that indicate the degree of their relevance. These weights are an example of feature alignment information. The mutual feature adjustment unit 25d then synthesizes these weights with the visual features to output the visual features VFa after feature alignment.

[0040] The mutual feature adjustment unit 25c, which corresponds to the text feature group, has basically the same configuration as the mutual feature adjustment unit 25d. However, in the mutual attention mechanism 31 of the mutual feature adjustment unit 25c, the text feature group is input as query q, and the visual features are input as key k and value v. The conversion unit 33, which corresponds to the text feature group, is the same as the conversion unit 34, which corresponds to the visual features. The conversion units 33 and 34 can perform, for example, the following conversions. Linear transformation • Downsampling → Upsampling Linear transformation → ReLU (Rectified Linear Unit) function → Linear transformation → Sigmoid function • MLP (Multilayer Perceptron) No conversion

[0041] Figure 9 shows a flowchart of the processing performed by the mutual feature adjustment unit 25. Note that the flowchart in Figure 9 represents the processing performed by the mutual feature adjustment unit 25d corresponding to the visual features shown in Figure 8, in the configuration shown in Figure 7. First, the mutual feature adjustment unit 25d receives visual features from the visual feature adjustment unit 24a (step S11). Next, the mutual feature adjustment unit 25d, using the mutual attention mechanism 32, refers to the text feature group received from the text feature adjustment unit 23a (step S12), emphasizes visual features highly related to the text feature group, and generates feature alignment information (step S13). Then, the mutual feature adjustment unit 25d synthesizes the feature alignment information with the visual features (step S14) and outputs it as the visual feature VFa after mutual feature alignment (step S15).

[0042] The processing of the mutual feature adjustment unit 25c corresponding to the text features shown in Figure 8 is basically the same. However, the mutual feature adjustment unit 25c acquires text features in step S11, generates feature alignment information by emphasizing text features highly related to visual features in steps S12 and S13, and synthesizes the feature alignment information with the text features in step S14.

[0043] Next, variations of the mutual feature adjustment unit 25 will be described. Figure 10 shows the functional configuration of the mutual feature adjustment unit 25x related to the first variation. In Figure 10, the mutual attention mechanism 32a is the same as the mutual attention mechanism 32 included in the mutual feature adjustment unit 25d in Figure 8. Also, the mutual attention mechanism 32b is the same as the mutual attention mechanism included in the mutual feature adjustment unit 25c in Figure 8, where the text feature group is input to query q and the visual features are input to key k and value v. In the first variation, the mutual feature adjustment unit 25x merges the output of the mutual attention mechanism 32a corresponding to the text feature group and the output of the mutual attention mechanism 32b corresponding to the visual features and inputs it to the conversion unit 35. The merging is performed by methods such as averaging, addition, and pooling.

[0044] The transformation unit 35 is composed of a network with learnable parameters. The output of the transformation unit 35 is input to transformation unit 34a, which corresponds to the text feature group, and transformation unit 34b, which corresponds to the visual feature group. Transformation units 34a and 34b are identical to the transformation unit 34 shown in Figure 8. Transformation unit 34a generates alignment information ALT corresponding to the text feature group based on the output from transformation unit 35. Transformation unit 34b generates alignment information ALv corresponding to the visual feature group based on the output from transformation unit 35.

[0045] Figure 11(A) shows the functional configuration of the mutual feature adjustment unit 25y related to the second variation. The mutual feature adjustment unit 25y related to the second variation fuses a group of text features into a single text feature. Fusion is performed by methods such as averaging, addition, and pooling. The fused text feature is input to the channel-wise MLP 36. The channel-wise MLP is a fully connected layer that processes each channel, and processes the fused text feature channel by channel before outputting the result. The mutual feature adjustment unit 25y synthesizes the output of the channel-wise MLP 36 with the visual features to generate alignment information ALv corresponding to the visual features.

[0046] Figure 11(B) shows the functional configuration of the mutual feature adjustment unit 25z related to the third variation. The mutual feature adjustment unit 25z related to the third variation also fuses a group of text features into a single text feature. The fused text feature is input to the channel-wise MLP 36. The channel-wise MLP is a fully connected layer that processes each channel, and processes the fused text feature channel by channel before outputting the result. Meanwhile, the visual features are input to the MLP 37. The mutual feature adjustment unit 25z generates alignment information ALT corresponding to the text features and alignment information ALv corresponding to the visual features by combining the outputs of the channel-wise MLP 36 and MLP 37.

[0047] (Learning structure) Next, the configuration of the information processing device 100 during learning will be described. Figure 12 shows the configuration of the information processing device 100 during learning. In the example shown in Figure 12, the mutual feature adjustment unit 25 performs mutual feature alignment using the intermediate features of the text feature adjustment unit 23 and the visual feature adjustment unit 24.

[0048] The information processing device 100x used during learning includes, in addition to the inference components shown in Figure 3, a data storage unit 5 for storing learning data and a learning unit 28. The data storage unit 5 stores visual information (images or videos) to be classified and corresponding text groups, which are the correct answer information, as learning data. During learning, the visual information VI included in the learning data is input to the visual feature extraction unit 22, and the text group TX, which is the correct answer information for that visual information, is input to the text feature extraction unit 21.

[0049] The text feature extraction unit 21 extracts text features from each text included in the text group TX and outputs them to the text feature adjustment unit 23 as a text feature group TF. The visual feature extraction unit 22 extracts visual features VF from the visual information VI and outputs them to the visual feature adjustment unit 24. The mutual feature adjustment unit 25 generates feature alignment information showing the interrelationship between the text feature group and the visual features based on the intermediate features of the text feature group obtained from the text feature adjustment unit 23 and the intermediate features of the visual features obtained from the visual feature adjustment unit 24, and outputs it to the text feature adjustment unit 23 and the visual feature adjustment unit 24, respectively.

[0050] The text feature adjustment unit 23 performs feature alignment of the text feature group using the input feature alignment information and outputs the text feature TFa after feature alignment to the classification unit 26. The visual feature adjustment unit 24 performs feature alignment of the visual features using the input feature alignment information and outputs the visual feature VFa after feature alignment to the classification unit 26. The classification unit 26 classifies the visual feature VFa based on the similarity between the visual feature VFa and the text feature group TFa and outputs the classification result to the learning unit 28.

[0051] The learning unit 28 optimizes the text feature adjustment unit 23, the visual feature adjustment unit 24, and the mutual feature adjustment unit 25 based on the classification results. Specifically, the learning unit 28 optimizes the parameters of the networks that constitute the text feature adjustment unit 23, the visual feature adjustment unit 24, and the mutual feature adjustment unit 25. In this way, during learning, the text feature adjustment unit 23, the visual feature adjustment unit 24, and the mutual feature adjustment unit 25 are optimized using the learning data. Thus, a pre-trained information processing device 100 is obtained.

[0052] [Classification process] Next, the classification process performed by the information processing device 100 described above will be explained. Figure 13 is a flowchart of the classification process. This process is realized when the processor 11 shown in Figure 2 executes a pre-prepared program and operates as each component shown in Figure 3, etc.

[0053] First, the information processing device 100 acquires visual information VI and text data TX (step S20). Next, the information processing device 100 performs the processing of the visual information and the processing of the text data in parallel. First, the processing of the visual information will be explained.

[0054] The visual feature extraction unit 22 extracts visual features VF from visual information VI (step S21a). Next, the visual feature adjustment unit 24 performs feature alignment of the visual features (step S22a). Next, the mutual feature adjustment unit 25 refers to the text feature group (step S23a), emphasizes the visual features that are highly related to the text feature group to generate feature alignment information (step S24a), and outputs the feature alignment information to the visual feature adjustment unit 24 (step S25a). Next, the visual feature adjustment unit 24 uses the input feature alignment information to perform feature alignment of the visual features VF and outputs the feature-aligned visual features VFa to the classification unit 26 (step S26a).

[0055] Next, the processing of the text group will be explained. The processing of the text group is basically the same as the processing of visual information. First, the text feature extraction unit 21 extracts the text feature group TF from the text group TX (step S21b). Next, the text feature adjustment unit 23 performs feature alignment on the text feature group TF (step S23b). Next, the mutual feature adjustment unit 25 refers to the visual features (step S23b), emphasizes the text feature groups that are highly related to the visual features to generate feature alignment information (step S24b), and outputs the feature alignment information to the text feature adjustment unit 23 (step S25b). Next, the text feature adjustment unit 23 performs feature alignment on the text feature group TF using the input feature alignment information and outputs the text feature group TFa after feature alignment to the classification unit 26 (step S26b).

[0056] Next, the classification unit 26 calculates the similarity between the input visual feature VFa and each text feature included in the input text feature group TFa (step S27), and then classifies the visual information (step S28). Specifically, the classification unit 26 determines that the text of the text feature with the highest similarity to the visual feature is the text corresponding to the visual information. The classification unit 26 may also output multiple texts as classification results in descending order of similarity. Then, the classification process ends.

[0057] [Verification Results] Next, the verification results of the classification process by the information processing device 100 of this embodiment will be explained. Figure 14 shows the results of the 5-class classification by the information processing device 100. "Number of training data per class" is the number of training data for each class used during training. "Comparative Example 1" shows an example in which feature alignment was not performed on the visual features and text feature groups. That is, Comparative Example 1 shows the case in Figure 3 where the text feature adjustment unit 23, the visual feature adjustment unit 24, and the mutual feature adjustment unit 25 are omitted, and the classification unit 26 performs classification based on the text feature group TF and the visual feature VF. "Comparative Example 2" shows an example in which feature alignment was performed independently on the visual features and the text feature groups, but mutual feature alignment was not performed. That is, Comparative Example 1 shows the case in Figure 3 where the mutual feature adjustment unit 25 is omitted, and the classification unit 26 performs classification based on the text feature group TFa and the visual feature VFa.

[0058] As can be seen from Figure 14, the accuracy of the proposed method surpasses that of Comparative Examples 1 and 2 in all cases of training data size. Thus, the method of this embodiment enables more accurate classification by performing mutual feature alignment between text features and visual features.

[0059] Figure 15 shows feature maps representing the distribution of features in the feature space for Comparative Example 2 and the proposed method. Feature map 51 is the feature map for Comparative Example 2, and feature map 52 is the feature map for the proposed method. In each feature map, dots (·) represent visual features, and crosses (×) represent text features. This example is a 7-class classification example, and each class (0-6) in the feature map is color-coded.

[0060] In the feature map 51 of Comparative Example 2, the dots representing visual features are distributed in a somewhat clustered manner for each class. In contrast, the cross marks representing text features are concentrated in almost the same location. That is, in this example, the cross marks for 7 classes are in almost the same location (see circle 51x) and overlap. Thus, in Comparative Example 2, it can be seen that image features and text features are independently aligned in the feature space.

[0061] In the feature map 52 of the proposed method, the dots representing visual features are distributed in a certain degree of clustering for each class. Furthermore, the cross marks representing text features are distributed in a manner corresponding to the clusters of visual features corresponding to each class, as indicated by the individual circles 52x. In other words, according to the proposed method, image features and text features are aligned according to their interrelationships. Thus, the method of this embodiment makes it possible to perform feature alignment that takes into account the interrelationships between image features and text features.

[0062] [Examples of application] The information processing device disclosed herein can be applied, for example, to the management of the actions of people, robots, and other entities in industrial settings. Specifically, the method disclosed herein can be used for automating warehouses in the logistics industry, improving the efficiency of stores in the retail industry, improving the efficiency of site management in the construction industry, and automating inspections in the manufacturing industry.

[0063] Figure 16 shows an example of an activity management system to which the information processing device of the present disclosure is applied. The activity management system 200 comprises a camera 210, an activity recognition device 220, and a management database 230. The camera 210 is installed at the site to be managed and captures images and videos of the site and transmits them to the information processing device 100. The activity recognition device 220 is configured using the above-mentioned information processing device 100 and classifies and recognizes the actions and tasks of people working at the site based on the visual information obtained by the camera 210. The activity recognition device 220 then associates the actions of each recognized person with the time and location at the site and records them in the management database 230 as an activity history. As a result, the site manager can manage workers based on the activity history of each person recorded in the management database 230.

[0064] <Second Embodiment> Figure 17 is a block diagram showing the functional configuration of an information processing device according to another example of the present disclosure. The information processing device 70 comprises a visual feature extraction means 71, a text feature extraction means 72, and a feature adjustment means 73.

[0065] Figure 18 is a flowchart of the processing performed by the information processing device 70. The visual feature extraction means 71 extracts visual features from the target visual information (step S71). The text feature extraction means 72 extracts text features from the text corresponding to the visual features (step S72). The feature adjustment means 73 performs feature alignment of the text features and the visual features based on the interrelationship between the text features and the visual features (step S73).

[0066] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0067] (Note 1) A visual feature extraction method for extracting visual features from visual information, A text feature extraction means for extracting text features from text corresponding to the aforementioned visual features, A feature adjustment means that performs feature alignment of the text features and the visual features based on the interrelationship between the text features and the visual features, An information processing device equipped with the following features.

[0068] (Note 2) The feature adjustment means is an information processing device according to Appendix 1, which adjusts the text features and the visual features on the same feature space.

[0069] (Note 3) The aforementioned feature adjustment means is A text feature adjustment means for performing feature alignment of the aforementioned text features, A visual feature adjustment means for performing feature alignment of the aforementioned visual features, A mutual adjustment means that reflects feature alignment information for performing feature alignment based on the interrelationship between the text features and the visual features in at least one of the text feature adjustment means and the visual feature adjustment means, An information processing device as described in Appendix 1 or 2, comprising:

[0070] (Note 4) The mutual adjustment means is an information processing device as described in Appendix 3, which determines the interrelationship between the text features and the visual features based on the respective inputs of the text feature adjustment means and the visual feature adjustment means, or the respective outputs of the text feature adjustment means and the visual feature adjustment means, or the respective intermediate features of the text feature adjustment means and the visual feature adjustment means.

[0071] (Note 5) The mutual adjustment means determines the relationship between the text features and the visual features using a mutual attention mechanism and generates the feature alignment information, as described in Appendix 3 or 4.

[0072] (Note 6) The text feature extraction means is an information processing device according to any one of the appendices 1 to 5 that generates the text feature by fusing features extracted from multiple texts.

[0073] (Note 7) An information processing apparatus according to any one of the appendices 1 to 7, comprising a classification means for classifying the visual information into one of a plurality of texts based on the similarity between the text features after feature alignment and the visual features after feature alignment.

[0074] (Note 8) The aforementioned visual information is information capturing the behavioral state of a person. The aforementioned text describes the actions of a person. The classification means is an information processing device according to Appendix 7 that recognizes the actions of a person included in the visual information.

[0075] (Note 9) A method of information processing performed by a computer, Extract visual features from visual information, Text features are extracted from the text corresponding to the aforementioned visual features. An information processing method that performs feature alignment of the text features and the visual features based on the interrelationship between the text features and the visual features.

[0076] (Note 10) Extract visual features from visual information, Text features are extracted from the text corresponding to the aforementioned visual features. A program that causes a computer to perform a feature alignment of the text features and the visual features based on the interrelationship between the text features and the visual features.

[0077] Furthermore, some or all of the configurations described in Appendices 2 to 8, which are subordinate to Appendice 1 above, may also be subordinate to Appendices 9 and 10 in the same way as those described in Appendices 2 to 8. Moreover, not limited to Appendices 1, 9, and 10, some or all of the configurations described as appendices may also be subordinate to various hardware, software, various recording means for recording software, or systems, without departing from the embodiments described above.

[0078] Although the present disclosure has been described above with reference to embodiments and examples, the present disclosure is not limited to the above embodiments and examples. Various modifications to the structure and details of the present disclosure can be understood by those skilled in the art within the scope of the present disclosure. [Explanation of Symbols]

[0079] 11 processors 21 Text Feature Extraction Unit 22 Visual Feature Extraction Unit 23 Text Feature Adjustment Unit 24 Visual Feature Adjustment Unit 25 Mutual Feature Adjustment Unit 26 Classification Department 28 Learning Department 100 Information Processing Devices

Claims

1. A visual feature extraction method for extracting visual features from visual information, A text feature extraction means for extracting text features from text corresponding to the aforementioned visual features, A feature adjustment means that performs feature alignment of the text features and the visual features based on the interrelationship between the text features and the visual features, An information processing device equipped with the following features.

2. The information processing apparatus according to claim 1, wherein the feature adjustment means adjusts the text features and the visual features on the same feature space.

3. The aforementioned feature adjustment means is A text feature adjustment means for performing feature alignment of the aforementioned text features, A visual feature adjustment means for performing feature alignment of the aforementioned visual features, A mutual adjustment means that reflects feature alignment information for performing feature alignment based on the interrelationship between the text features and the visual features in at least one of the text feature adjustment means and the visual feature adjustment means, The information processing apparatus according to claim 1, comprising:

4. The information processing apparatus according to claim 3, wherein the mutual adjustment means determines the interrelationship between the text features and the visual features based on the respective inputs of the text feature adjustment means and the visual feature adjustment means, or the respective outputs of the text feature adjustment means and the visual feature adjustment means, or the respective intermediate features of the text feature adjustment means and the visual feature adjustment means.

5. The information processing apparatus according to claim 3, wherein the mutual adjustment means determines the relationship between the text features and the visual features using a mutual attention mechanism and generates the feature alignment information.

6. The information processing apparatus according to claim 1, wherein the text feature extraction means generates the text feature by fusing features extracted from a plurality of texts.

7. The information processing apparatus according to claim 1, further comprising a classification means for classifying the visual information into one of a plurality of texts based on the similarity between the text features after feature alignment and the visual features after feature alignment.

8. The aforementioned visual information is information capturing the behavioral state of a person. The aforementioned text describes the actions of a person. The information processing apparatus according to claim 7, wherein the classification means recognizes the actions of a person included in the visual information.

9. A method of information processing performed by a computer, Extract visual features from visual information, Text features are extracted from the text corresponding to the aforementioned visual features. An information processing method that performs feature alignment of the text features and the visual features based on the interrelationship between the text features and the visual features.

10. Extract visual features from visual information, Text features are extracted from the text corresponding to the aforementioned visual features. A program that causes a computer to perform a feature alignment of the text features and the visual features based on the interrelationship between the text features and the visual features.

Images