Information processing device

Abstract

An information processing device according to the present invention comprises: a feature generation unit that generates feature amounts from each of a plurality of recognition target images obtained from a plurality of viewpoints; and an information integration unit that performs integration processing on the plurality of feature amounts generated by the feature generation unit.

Description

Information processing device

[0001] The present invention relates to a recognition technology using an image.

[0002] In recent years, the accuracy of action recognition tasks, which recognize human actions captured in video, has improved with the development of deep learning technologies such as convolutional neural networks (CNNs) (Non-Patent Document 1) and Transformers (Non-Patent Document 2). However, since most of these action recognition technologies recognize actions from images captured by a single camera, it is known that accuracy decreases in scenes where occlusion occurs due to obstacles or crowds (Non-Patent Document 7).

[0003] To mitigate the effects of occlusion, multi-view activity recognition is known, which simultaneously uses multiple views taken from different angles by multiple cameras to recognize actions. Multi-view activity recognition can significantly reduce the occlusion area by installing cameras at multiple angles, such as cameras installed in the four corners of a room, and is therefore expected to achieve good accuracy even in scenes where occlusion occurs due to obstacles or crowds.

[0004] In multi-view action recognition, the information required for action recognition is scattered across each view, so information integration between views is important. In the technology disclosed in Non-Patent Document 3, a feature extractor called a Video Encoder is used to extract features from a single view of an image, and an embedding vector is obtained separately for each view. Then, information integration between views is performed using weighting by a Conditional Random Field from the embedding vectors of multiple views.

[0005] In the technology disclosed in Non-Patent Document 4, similarly, information between views is integrated using a Transformer with a mechanism called Self-Attention (Non-Patent Document 2) from the embedding vectors of multiple views obtained by a Video Encoder. Here, Self-Attention is a mechanism that extracts features by calculating a weighted sum of features based on the similarity between input sequences.

[0006] J. Carreira, et al., "Quo vadis, action recognition? A new model and the kinetics dataset," in CVPR, 2017.A. Vaswani, et al., "Attention is all you need," NIPS, 2017.D. Wang, et al., "Dividing and aggregating network for multi-view action recognition," in ECCV, 2018.M. Yasuda, et al., "Multi-view and multi-modal event detection utilizing transformer-based multi-sensor fusion," in ICASSP, 2022.Y. Hou, et al., "Multiview detection with feature perspective transformation," in ECCV, 2020.S. Ren, et al., "Faster r-cnn: Towards real-time object detection with region proposal networks," in NIPS, 2015.Q. Kong, et al., "Mmact: A large-scale dataset for cross modal human action understanding," in ICCV, 2019.

[0007] Multi-view action recognition demonstrates superior recognition accuracy compared to techniques using only a single view in occluded scenes. However, conventional multi-view action recognition integrates information across views using a single embedding vector representing the entire video for each view. Therefore, it can only recognize one action per person or per scene, and cannot simultaneously recognize the actions of multiple people. This issue can arise not only when the recognition target is a person, but also in other situations.

[0008] The present invention has been made in consideration of the above points, and aims to provide a technology that enables recognition processing of multiple recognition targets based on images obtained from multiple viewpoints.

[0009] According to the disclosed technology, there is provided an information processing device including: a feature generation unit that generates feature quantities from each of a plurality of recognition target images obtained from a plurality of viewpoints; and an information integration unit that performs integration processing on the plurality of feature quantities generated by the feature generation unit.

[0010] The disclosed technology provides a technology that enables recognition processing of multiple recognition targets based on images obtained from multiple viewpoints.

[0011] FIG. 1 is a configuration diagram of an information processing device 100. FIG. 1 is a configuration diagram of the information processing device 100 in a first embodiment. FIG. 2 is a flowchart showing the overall processing flow of the information processing device 100. FIG. 3 is a flowchart showing the processing flow of the video feature generation unit 1. FIG. 4 is a flowchart showing the processing flow of the person feature generation unit 2. FIG. 5 is a flowchart showing the processing flow of the information integration unit 3. FIG. 6 is a flowchart showing the processing flow of the prediction generation unit 4. FIG. 7 is a configuration diagram of the information integration unit 3 in the first embodiment. FIG. 8 is a flowchart showing the processing flow of the information integration unit 3 in the first embodiment. FIG. 9 is a configuration diagram of the information integration unit 3 in a second embodiment. FIG. 10 is a flowchart showing the processing flow of the information integration unit 3 in the second embodiment. FIG. 11 is a configuration diagram of the information integration unit 3 in a third embodiment. FIG. 12 is a flowchart showing the processing flow of the information integration unit 3 in the third embodiment. FIG. 13 is a diagram showing an example of the hardware configuration of the information processing device 100.

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0013] In the following description, the recognition target is a person, but the recognition target is not limited to a person. The technology according to the present invention can be applied to any recognition target.

[0014] Furthermore, there is no particular limitation on the type of recognition to be performed. The recognition according to the present invention may be any classification task, such as behavior recognition, age determination, gender determination, or belongings recognition. For example, the traveling direction of a recognition target can be estimated if this is a classification task.

[0015] Furthermore, since the "video" in this embodiment is a collection of images, the "video" may also be called "images."

[0016] (Outline of the embodiment) An outline of the configuration and operation of an information processing device 100 according to the present embodiment will be described. The information processing device 100 according to the present embodiment recognizes the actions of multiple people from multi-view video. That is, while conventional technology has been able to recognize only one action from multi-view video, the information processing device 100 according to the present embodiment can recognize the actions of multiple people.

[0017] More specifically, the information processing device 100 recognizes the actions of multiple people using multi-view video and bounding boxes that represent the positions of people within the video. Here, the bounding boxes are assumed to be linked to the same person across multiple views by IDs. Note that the image (or feature map) of the portion surrounded by the bounding box may be referred to as the recognition target image.

[0018] Fig. 1 shows an example of the configuration of an information processing device 100 according to an embodiment of the present invention. As shown in Fig. 1, the information processing device 100 includes a video feature generation unit 1, a person feature generation unit 2, an information integration unit 3, and a prediction generation unit 4. An overview of the operation of the information processing device 100 having this configuration will be described below.

[0019] First, the video feature generation unit 1 (e.g., a video encoder) extracts a feature map representing the entire video for each view. Here, a feature map is a feature quantity with spatial resolution (spatial information), and information about a specific location in the video is contained in the corresponding location in the feature map. For example, the hand position in the feature map contains hand information, and the foot position contains foot information.

[0020] Next, person feature generation unit 2 (e.g., RoIAlign (Non-Patent Document 6)) extracts a feature map of each person from the feature map of the entire video for each view. RoIAlign, an example of a mechanism used as person feature generation unit 2, is a mechanism that performs processing to cut out a bounding box area of ​​a fixed size that represents the position of a person in the video from the feature map. Note that person feature generation unit 2 may also be called a feature generation unit. Furthermore, "video feature generation unit 1 and person feature generation unit 2" may also be called a feature generation unit.

[0021] Furthermore, the processing of the person feature generation unit 2 is not limited to processing using RoIAlign, and any means may be used as long as it is capable of extracting a bounding box area from a feature map.

[0022] Then, an information integration unit 3 (e.g., Transformer) integrates information between views based on feature maps of the same person in each view. Note that using a Transformer as the information integration unit 3 is just one example. Any means may be used as long as it can integrate information from multiple views.

[0023] In multi-view recognition tasks where there are multiple recognition targets, such as multi-view object detection (Non-Patent Document 5), it is important to utilize spatial information in information integration between views. However, in the conventional multi-view action recognition techniques described in Non-Patent Documents 1 and 2, information integration is performed using embedding vectors that have lost spatial information, and therefore spatial information is not effectively utilized.

[0024] Therefore, in this embodiment, unlike the conventional technology, information integration is performed using a feature map having spatial information instead of an embedding vector, which is expected to enable highly accurate recognition.

[0025] Furthermore, the Transformer, which is an example of a mechanism used as the information integration unit 3 in this embodiment, differs from a typical Transformer that is composed of Self-Attention, which calculates similarities between feature maps of all views and extracts features, by using Same View Attention (SVA), which extracts features from similarities only between feature maps of the same view, and Different View Attention (DVA), which extracts features from similarities only between feature maps of different views.

[0026] The above two mechanisms (SVA and DVA) enable more powerful information integration between views than Self-Attention. Note that it is also possible to use a normal Transformer with Self-Attention as the information integration unit 3.

[0027] Finally, the information integration unit 3 (Transformer) integrates information between multiple views, and the resulting feature values ​​are input to the prediction generation unit 4 (classifier), which then produces a prediction of the behavior.

[0028] The information processing device 100 may be a device having any one or more of the "video feature generation unit 1, person feature generation unit 2, information integration unit 3, and prediction generation unit 4." For example, the video feature generation unit 1, person feature generation unit 2, information integration unit 3, and prediction generation unit 4 may each be separate information processing devices 100 (i.e., computers), or the information processing device 100 may be a device having any two or more of these four functional units.

[0029] Hereinafter, first to third embodiments will be described as more specific embodiments of the information processing device 100.

[0030] (First embodiment) <Device configuration and operation> First, the first embodiment will be described. Fig. 2 is a configuration diagram of an information processing device 100 in the first embodiment. As shown in Fig. 2, the information processing device 100 in the first embodiment has a video feature generation unit 1, a person feature generation unit 2, an information integration unit 3, a prediction generation unit 4, a video storage unit 5, a video feature generation parameter storage unit 6, a person position storage unit 7, an information integration parameter storage unit 8, and a prediction generation parameter storage unit 9.

[0031] In addition, any one, any plurality, or all of the "video storage unit 5, video feature generation parameter storage unit 6, person position storage unit 7, information integration parameter storage unit 8, and prediction generation parameter storage unit 9" may be provided outside the information processing device 100.

[0032] 2 shows multiple video feature generation units 1 and multiple person feature generation units 2 to clearly show that operations are performed for each view, but there is only one video feature generation unit 1 and one person feature generation unit 2 each that processes multiple views. Note that a video feature generation unit 1 and a person feature generation unit 2 may be provided for each view. In other words, the same number of video feature generation units 1 and person feature generation units 2 as the number of views may be provided.

[0033] The overall operation of the information processing device 100 will be described with reference to FIG.

[0034] Video feature generation unit 1 (e.g., Video Encoder) acquires video feature generation parameters from video feature generation parameter storage unit 6, performs video feature generation based on the parameters for multiple views of input video input from video storage unit 5, and generates a feature map representing the entire video for each view. These video feature generation parameters are parameters of a neural network model, and are obtained by supervised learning of a general neural network using training data for a target task.

[0035] The person feature generation unit 2 (e.g., RoIAlign) generates a person feature map by cutting out the feature map generated by the video feature generation unit 1 based on the bounding box representing the position of a person in each view and the ID representing the same person, which are input from the person position storage unit 7.

[0036] The information integration unit 3 (e.g., Transformer) acquires information integration parameters from the information integration parameter storage unit 8, and performs information integration based on the parameters on the person feature maps of each view of the same person obtained by the person feature generation unit 2, to generate a feature map that integrates information from multiple views. These information integration parameters are parameters of a neural network model, and are obtained by supervised learning of a general neural network using training data for the target task.

[0037] The prediction generation unit 4 acquires prediction generation parameters from the prediction generation parameter storage unit 9 and performs parameter-based prediction on the feature quantities output by the information integration unit 3. These prediction generation parameters are parameters of a neural network model and are obtained by supervised learning of a general neural network using training data for the target task.

[0038] <Overall Processing Flow> Fig. 3 is a flowchart showing the overall processing flow of the information processing device 100 according to this embodiment. The overall processing flow will be described with reference to Fig. 3 .

[0039] In S1 (step 1 ), the video feature generation unit 1 acquires, from the video feature generation parameter storage unit 6 , parameters for video feature generation stored in the video feature generation parameter storage unit 6 .

[0040] In S2, the video feature generation unit 1 acquires input videos of multiple views from the video storage unit 5.

[0041] In S3, the image feature generation unit 1 performs image feature generation for the input image of each view using the image feature generation parameters.

[0042] In S4 , the person feature generation unit 2 acquires the bounding box and ID stored in the person position storage unit 7 from the person position storage unit 7 .

[0043] In S5 , the image feature generation unit 1 transmits the generated image feature map to the person feature generation unit 2 .

[0044] In S6, the person feature generation unit 2 performs a cutout process based on the bounding box and ID for the video feature amount of each view, thereby generating person features.

[0045] In S7 , the information integration unit 3 acquires, from the information integration parameter storage unit 8 , parameters for information integration stored in the information integration parameter storage unit 8 .

[0046] In S8 , the person feature generation unit 2 transmits the generated person feature map to the information integration unit 3 .

[0047] In S9, the information integration unit 3 generates a feature map by performing information integration between views using the information integration parameters on the person feature map.In S10, the prediction generation unit 4 acquires parameters for prediction generation stored in the prediction generation parameter storage unit 9 from the prediction generation parameter storage unit 9.

[0048] In S11, the information integration unit 3 transmits the generated feature quantities to the prediction generation unit 4. In S12, the prediction generation unit 4 performs prediction generation using the prediction generation parameters for the feature quantities received from the information integration unit 3. In S13, the prediction generation unit 4 outputs the generated prediction result.

[0049] In the above process, parameters are acquired for each process in the video feature generation unit 1, the person feature generation unit 2, the information integration unit 3, and the prediction generation unit 4, but instead, the parameters of each unit may be acquired together in advance.

[0050] The operation of each unit constituting the information processing device 100 will be described in detail below with reference to flowcharts.

[0051] <Video Feature Generation Unit 1> The operation of the video feature generation unit 1 will be described with reference to the flowchart of FIG.

[0052] In S101, the video feature generation unit 1 acquires video feature generation parameters from the video feature generation parameter storage unit 6. When processing a plurality of videos, it is sufficient to acquire these parameters only once, at the beginning.

[0053] In S102, the video feature generation unit 1 acquires input video of a plurality of views from the video storage unit 5. Here, the video of each view is acquired as a four-dimensional vector of "number of frames x height x width x RGB".

[0054] In S103, the video feature generation unit 1 applies video feature generation based on the model parameters to each of the multiple videos acquired in S102.

[0055] In S104 , the image feature generation unit 1 transmits the image feature map of the multiple views generated in S103 to the person feature generation unit 2 .

[0056] <Personal Feature Generation Unit 2> Next, the operation of the personal feature generation unit 2 will be described with reference to the flowchart of FIG.

[0057] In S201, the person feature generation unit 2 acquires, from the person position storage unit 7, the coordinates of a bounding box that indicates the position of a person in the video of each view, and an ID that indicates the same person as the person.

[0058] In S202, person feature generation unit 2 acquires video feature maps of a plurality of views from video feature generation unit 1.

[0059] In S203, the person feature generation unit 2 extracts the bounding box area acquired in S201 from the video feature map acquired in S202 and generates a person feature map for each person by linking the same person by ID. Here, the person feature map for one person in each view is acquired as a three-dimensional vector of "height x width x channel."

[0060] In S204, the person feature generation unit 2 outputs the person feature map generated in S203 to the information integration unit 3.

[0061] <Information Integration Unit 3> Next, the operation of the information integration unit 3 will be described with reference to the flowchart of FIG.

[0062] In S301, the information integration unit 3 acquires information integration parameters from the information integration parameter storage unit 8. When processing a plurality of videos, it is sufficient to acquire these parameters only once, at the beginning.

[0063] In S302 , the information integration unit 3 acquires the person feature map from the person feature generation unit 2 .

[0064] In S303, the information integration unit 3 performs information integration between views based on model parameters for feature maps of the same ID (same person) among the multiple person feature maps acquired in S302, and generates a feature amount that integrates the information of the multiple views.

[0065] In S304, the information integration unit 3 outputs the feature quantity generated in S303 to the prediction generation unit 4.

[0066] <Prediction Generation Unit 4> Next, the operation of the prediction generation unit 4 will be described with reference to the flowchart of FIG.

[0067] In S401, the prediction generation unit 4 acquires prediction generation parameters from the prediction generation parameter storage unit 9. When processing a plurality of videos, it is sufficient to acquire these parameters only once, at the beginning.

[0068] In S402 , the prediction generation unit 4 acquires the feature amount from the information integration unit 3 .

[0069] In S404, the prediction generation unit 4 predicts a desired task based on the model parameters for the feature amount acquired in S402, and outputs the prediction of S403.

[0070] <Details of Information Integration Unit 3> Next, a description will be given of details of the information integration unit 3. FIG.

[0071] As shown in FIG. 8 , the information integration unit 3 includes a position information addition unit 31 (corresponding to Positional Embedding), a view information addition unit 32 (corresponding to View Embedding), a same-view relationship extraction unit 33 (corresponding to Same View Attention), and a different-view relationship extraction unit 34 (corresponding to Different View Attention).

[0072] The position information adding unit 31 adds position information parameters to the feature map of each view to add information about the positional relationship of each pixel in the feature map. The view information adding unit 32 adds view information parameters to the feature map of each view to add information about which view the feature map belongs to.

[0073] The same-view relation extraction unit 33 extracts relations based on similarity between pixels in the feature maps of each view only among pixels within that feature map, while the different-view relation extraction unit 34 extracts relations based on similarity between pixels in the feature maps of different views only among pixels in the feature maps of each view.

[0074] Furthermore, the information integration unit 3 is made up of L layers (for example, L=4) when a pair consisting of one same-view relation extraction unit 33 and one different-view relation extraction unit 34 is considered as one layer. Furthermore, although the description here is based on relationship extraction based on similarity such as attention, other nonlinear processing using a neural network may also be used.

[0075] The detailed operation of the information integration unit 3 will be described with reference to the flowchart of FIG.

[0076] In S501, the information integration unit 3 acquires information integration parameters from the information integration parameter storage unit 8. When processing multiple videos, it is sufficient to acquire these parameters only once at the beginning. The information integration parameters include position information parameters, view information parameters, same-view relationship extraction parameters, and different-view relationship extraction parameters.

[0077] In S502, the information integration unit 3 acquires a person feature map for each view from the person feature generation unit 2.

[0078] In S503, the position information adding unit 31 adds the position information parameters to the person feature map of each view acquired in S502, and updates the person feature map of each view.

[0079] In S504, the view information adding unit 32 adds the view information parameters to the person feature map of each view acquired in S503, and updates the person feature map of each view.

[0080] In S505, the first-layer same-view relation extraction unit 33 updates the person feature map of each view acquired in S504 from the relationship between pixels of the feature map of the same view based on the model parameters.

[0081] In S506, the first-layer different view relation extraction unit 34 updates the person feature map of each view acquired in S505 from the relationship between pixels of the feature maps of different views based on the model parameters.

[0082] In S507, the processes of S505 and S506 are repeated for the number of layers to update the person feature map.

[0083] In S508, the information integration unit 3 outputs the person feature map of the Lth layer to the prediction generation unit 4 as a feature amount obtained by integrating information from a plurality of views.

[0084] Second Embodiment Next, a second embodiment will be described. In the second embodiment, the configuration and operation of the information integration unit 3 are different from those in the first embodiment.

[0085] In the first embodiment, in the information integrating unit 3, the feature map is updated by processing by the same-view relation extracting unit 33, and then the feature map is further updated by processing by the different-view relation extracting unit 34. In this case, the feature map input to the first-layer different-view relation extracting unit 34 does not include information on multiple views. Therefore, the processing by the first-layer same-view relation extracting unit 33 can be considered to be similar to the processing by the video feature generating unit 1 to generate features from video of a single view. Therefore, in the second embodiment, the information integrating unit 3 is configured as follows.

[0086] Fig. 10 shows the configuration of the information integration unit 3 in the second embodiment. As shown in Fig. 10, in the second embodiment, in the information integration unit 3, the feature map is updated by processing by the different view relation extraction unit 34, and then the feature map is further updated by processing by the same view relation extraction unit 33.

[0087] The detailed operation of the information integration unit 3 in the second embodiment will be described with reference to the flowchart of FIG.

[0088] In S601, the information integration unit 3 acquires information integration parameters from the information integration parameter storage unit 8. When processing multiple videos, it is sufficient to acquire these parameters only once at the beginning. The information integration parameters include position information parameters, view information parameters, parameters for extracting relationships between different views, and parameters for extracting relationships between identical views.

[0089] In S602, the information integration unit 3 acquires a person feature map for each view from the person feature generation unit 2.

[0090] In S603, the position information adding unit 31 adds the position information parameters to the person feature map of each view acquired in S602, and updates the person feature map.

[0091] In S604, the view information adding unit 32 adds the view information parameters to the person feature map of each view acquired in S603, and updates the person feature map.

[0092] In S605, the first-layer different view relation extraction unit 34 updates the person feature map of each view acquired in S604 from the relationship between pixels of the feature maps of different views based on the model parameters.

[0093] In S606, the first-layer same-view relation extraction unit 33 updates the person feature map of each view acquired in S605 from the relationship between pixels of the feature map of the same view based on the model parameters.

[0094] In S607, the processes of S605 and S606 are repeated for the number of layers to update the person feature map.

[0095] In S608, the information integration unit 3 outputs the person feature map of the Lth layer to the prediction generation unit 4 as a feature amount obtained by integrating information from a plurality of views.

[0096] Third Embodiment Next, a third embodiment will be described. In the third embodiment, the configuration and operation of the information integration unit 3 differs from those in the first and second embodiments.

[0097] In the first and second embodiments, as shown in FIG. 8 or FIG. 10 , the information integrating unit 3 arranges the same-view relation extraction unit 33 and the different-view relation extraction unit 34 in series, and updates the feature map by alternately repeating the processing.

[0098] Fig. 12 shows the configuration of the information integration unit 3 in the third embodiment. As shown in Fig. 12, in the third embodiment, the information integration unit 3 has a different view relation extraction unit 33 and an identical view relation extraction unit 34 arranged in parallel to branch the processing.

[0099] The detailed operation of the information integration unit 3 in the third embodiment will be described with reference to the flowchart of FIG.

[0100] In S701, the information integration unit 3 acquires information integration parameters from the information integration parameter storage unit 8. When processing multiple videos, it is sufficient to acquire these parameters only once at the beginning. The information integration parameters include position information parameters, view information parameters, parameters for extracting relationships between different views, and parameters for extracting relationships between identical views.

[0101] In S702, the information integration unit 3 acquires a person feature map for each view from the person feature generation unit 2.

[0102] In S703, the position information adding unit 31 adds the position information parameters to the person feature map of each view acquired in S702, and updates the person feature map.

[0103] In S704, the view information adding unit 32 adds the view information parameters to the person feature map of each view acquired in S703, and updates the person feature map.

[0104] In S705, the first-layer different view relation extraction unit 34 updates the person feature map of each view acquired in S704 from the relationship between pixels of the feature maps of different views based on the model parameters.

[0105] In S706, the first-layer same-view relation extraction unit 33 updates the person feature map of each view acquired in S704 from the relationship between pixels in the feature map of the same view based on the model parameters. Note that the processes of S705 and S706 can be performed in parallel.

[0106] In S707, the information integration unit 3 combines the person feature maps acquired in S705 and S706 to update the person feature map. That is, the information integration unit 3 combines the person feature map updated by the first-layer different view relation extraction unit 34 and the person feature map updated by the same view relation extraction unit 33 to update the person feature map.

[0107] In S708, the processes of S705, S706, and S707 are repeated for the number of layers to update the person feature map.

[0108] In S709, the information integration unit 3 outputs the person feature map of the Lth layer to the prediction generation unit 4 as a feature amount obtained by integrating information from a plurality of views.

[0109] (Hardware Configuration Example) The information processing device 100 described in this embodiment can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0110] That is, the information processing device 100 can be realized by using hardware resources such as a CPU and memory built into a computer to execute a program corresponding to the processing performed by the information processing device 100. The program can be recorded on a computer-readable recording medium (such as a portable memory) and can be saved or distributed. The program can also be provided via a network such as the Internet or email.

[0111] Fig. 14 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 14 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B. The computer may further include a GPU.

[0112] The program that realizes the processing on the computer is provided by a recording medium 1001, such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0113] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the information processing device 100 in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0114] (Summary, Effects, etc. of the Embodiments) As described above, the technology described in the present embodiments makes it possible to perform recognition processing on a plurality of recognition targets based on images obtained from a plurality of viewpoints.

[0115] The following additional notes are provided regarding the above-described embodiments.

[0116] <Additional Notes> (Additional Item 1) An information processing device comprising: a feature generation unit that generates a feature from each of a plurality of recognition target images obtained from a plurality of viewpoints; and an information integration unit that performs an integration process on the plurality of feature quantities generated by the feature generation unit. (Additional Item 2) The information processing device according to Additional Item 1, wherein the feature generation unit generates a feature map having spatial information as the feature quantities. (Additional Item 3) The information processing device according to Additional Item 1, wherein the information integration unit, in the integration process on the plurality of feature quantities, distinguishes between extracting a relationship between feature quantities from the same viewpoint and extracting a relationship between feature quantities from different viewpoints. (Additional Item 4) An information processing device comprising: an information integration unit that performs an integration process on a plurality of feature quantities corresponding to a plurality of viewpoints, wherein the information integration unit, in the integration process on the plurality of feature quantities, distinguishes between extracting a relationship between feature quantities from the same viewpoint and extracting a relationship between feature quantities from different viewpoints. (Supplementary Item 5) An information processing method executed by an information processing device, comprising: a feature generation step of generating a feature from each of a plurality of recognition target images obtained from a plurality of viewpoints; and an information integration step of integrating the plurality of feature quantities generated by the feature generation step. (Supplementary Item 6) An information processing method executed by an information processing device, comprising: an information integration step of integrating a plurality of feature quantities corresponding to a plurality of viewpoints, wherein the information integration step, in the integration process for the plurality of feature quantities, distinguishes between extracting a relationship between feature quantities from the same viewpoint and extracting a relationship between feature quantities from different viewpoints. (Supplementary Item 7) A program for causing a computer to function as the information processing device according to any one of Supplementary Items 1 to 4. (Supplementary Item 8) A non-transitory storage medium storing a program for causing a computer to function as the information processing device according to any one of Supplementary Items 1 to 4.

[0117] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0118] REFERENCE SIGNS LIST 1 Video feature generation unit 2 Person feature generation unit 3 Information integration unit 4 Prediction generation unit 5 Video storage unit 6 Video feature generation parameter storage unit 7 Person position storage unit 8 Information integration parameter storage unit 9 Prediction generation parameter storage unit 31 Position information addition unit 32 View information addition unit 33 Same view relationship extraction unit 34 Different view relationship extraction unit 100 Information processing device 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. An information processing device comprising: a feature generation unit that generates features from each of a plurality of recognition target images obtained from a plurality of viewpoints; and an information integration unit that performs integration processing on the plurality of features generated by the feature generation unit.

2. The information processing device according to claim 1, wherein the feature generation unit generates a feature map having spatial information as the feature amount.

3. The information processing device according to claim 1, wherein the information integration unit, in the integration process for the plurality of feature quantities, distinguishes between extracting relationships between feature quantities from the same viewpoint and extracting relationships between feature quantities from different viewpoints.

4. An information processing device comprising an information integration unit that performs integration processing on multiple feature amounts corresponding to multiple viewpoints, wherein the information integration unit, in the integration processing on the multiple feature amounts, distinguishes between extracting relationships between feature amounts from the same viewpoint and extracting relationships between feature amounts from different viewpoints.

Images