Information processing system, information processing method, and information processing program

Abstract

An information processing system (10) comprises at least one processor. When a first estimated motion of a first person (P1) appearing in a captured image matches a second estimated motion that is of a wearer (second person (P2)) of a mobile terminal (30) and that has been derived on the basis of a sensing result by a sensor (34) mounted on the mobile terminal (30), the processor (processing unit (15)) outputs, to an output device (16), multimodal information in which first information pertaining to the first person (P1) in the captured image is associated with second information acquired from the mobile terminal (30).

Description

Information processing system, information processing method, and information processing program

[0001] An embodiment of the present invention relates to an information processing system, an information processing method, and an information processing program.

[0002] Conventionally, techniques have been disclosed for recognizing a person from information detected by a sensor or beacon attached to the person, as well as techniques for image recognition of a person appearing in a captured image.

[0003] However, conventional techniques have not integrated and managed multiple types of information about a person.

[0004] JP 2024-32409 A JP 2012-150613 A

[0005] The present invention has been made in consideration of the above, and aims to provide an information processing system, an information processing method, and an information processing program that can integrate and manage multiple types of information related to a person.

[0006] An information processing system according to an embodiment includes at least one processor, and when a first estimated movement of a first person appearing in a captured image matches a second estimated movement of a person wearing the mobile device derived based on sensing results from a sensor mounted on the mobile device, the at least one processor outputs to an output device multimodal information that associates first information about the first person in the captured image with second information acquired from the mobile device.

[0007] FIG. 1 is an explanatory diagram of an example of an information management system according to an embodiment. FIG. 2 is an explanatory diagram of an example of an application form of the information management system according to an embodiment. FIG. 3 is a schematic diagram of an example of the data configuration of photography management information. FIG. 4 is a schematic diagram of an example of the data configuration of wearing management information. FIG. 5 is a schematic diagram of an example of the data configuration of personal management information. FIG. 6 is an explanatory diagram of an example of derivation of a first estimated motion. FIG. 7 is a schematic diagram of an example of the data configuration of association management information. FIG. 8 is a flowchart showing an example of the flow of information processing performed by the information processing system according to an embodiment. FIG. 9A is an explanatory diagram of a series of usage scenarios of the information processing system. FIG. 9B is an explanatory diagram of a series of usage scenarios of the information processing system. FIG. 9C is an explanatory diagram of a series of usage scenarios of the information processing system. FIG. 10 is an explanatory diagram of an example of an information management system according to an embodiment. FIG. 11 is an explanatory diagram of an example of an application form of the information management system according to an embodiment. FIG. 12 is a schematic diagram of an example of the data configuration of photography management information. FIG. 13 is a schematic diagram of an example of the data configuration of wearing management information. Fig. 14 is a schematic diagram of an example of the data configuration of personal management information, Fig. 15 is a flowchart showing an example of the flow of calculation processing of location information of a mobile terminal, Fig. 16 is a flowchart showing an example of the flow of information processing executed by the information processing system of the embodiment, and Fig. 17 is a hardware configuration diagram.

[0008] Hereinafter, embodiments of an information processing system and an information processing method will be described in detail with reference to the accompanying drawings.

[0009] First Embodiment FIG. 1 is an explanatory diagram of an example of an information management system 1 according to this embodiment.

[0010] The information management system 1 is a system for managing multiple types of information related to a person P in an integrated manner.

[0011] The information management system 1 includes an information processing system 10, one or more image capturing devices 20, and one or more mobile terminals 30. The information processing system 10, the image capturing devices 20, and the mobile terminals 30 are communicatively connected via a network NW or the like.

[0012] The photographing device 20 acquires photographed image data by photographing and transmits the photographed image data to the information processing system 10. The photographing device 20 sequentially transmits the photographed images in chronological order to the information processing system 10. That is, the photographing device 20 transmits video data consisting of multiple photographed image data to the information processing system 10. In the following description, the photographed image data may be simply referred to as a photographed image.

[0013] The portable terminal 30 is a portable terminal that is worn by the second person P2. In other words, the portable terminal 30 is a portable terminal that can be carried by the second person P2. The second person P2 is an example of the person P, and is the person P wearing the portable terminal 30.

[0014] FIG. 2 is an explanatory diagram of an example of an application form of the information management system 1 of this embodiment.

[0015] The image capturing device 20 is disposed so as to be able to capture an image of a predetermined space in real space. The predetermined real space is preferably a space in which a plurality of persons P may exist. The predetermined space is, for example, a space inside a building such as a factory, but is not limited to this space.

[0016] 2 shows an example in which the information management system 1 includes a plurality of image capture devices 20, namely, image capture devices 20A to 20C. The image capture angles and image capture directions of the plurality of image capture devices 20 are adjusted in advance so that at least some of the image capture angles do not overlap.

[0017] Returning to FIG. 1, the explanation will be continued.

[0018] In this embodiment, a person P who is a target of management by the information management system 1 is wearing a mobile terminal 30. The person P wearing the mobile terminal 30 will be described as a second person P2. That is, the second person P2 is the person wearing the mobile terminal 30. The second person P2 is an example of a person P. Also, in this embodiment, a person P who appears in a captured image captured by the imaging device 20 will be described as a first person P1. Therefore, when a person P who exists in real space appears in a captured image, the person P who appears in the captured image will be referred to as a first person P1, and when the person P is wearing a mobile terminal 30, the person P will be described as a second person P2. Therefore, the first person P1 and the second person P2 may be the same person P.

[0019] The mobile terminal 30 includes a communication unit 31, a UI (user interface) unit 32, a storage unit 33, a sensor 34, a drive unit 35, and a processing unit 36. The communication unit 31, the UI unit 32, the storage unit 33, the sensor 34, the drive unit 35, and the processing unit 36 ​​are connected to each other so as to be able to communicate with each other via a bus or the like.

[0020] The communication unit 31 is a communication interface that communicates with the information processing system 10 via the network NW. The UI unit 32 has an input mechanism that accepts various operations by a user and an output mechanism that outputs various information. The input mechanism is, for example, an input device such as a touch panel, a keyboard, or a button. The output mechanism is, for example, a display that displays various information, a speaker that outputs various sounds, etc.

[0021] The storage unit 33 stores various types of data. The storage unit 33 is, for example, a random access memory (RAM), a semiconductor memory element such as a flash memory, a hard disk, an optical disk, etc. The storage unit 33 may be a storage device provided outside the mobile terminal 30. The storage unit 33 may also be a storage medium that stores or temporarily stores programs and various types of information downloaded via a local area network (LAN), the Internet, etc.

[0022] The sensor 34 senses information about the second person P2 and outputs sensing result information representing the sensing result to the processing unit 36. In this embodiment, the information representing the sensing result will be referred to as the sensing result or sensing result information. The sensing result information includes at least the sensing result of the sensor 34 and the terminal ID of the mobile terminal 30 on which the sensor 34 is mounted. The sensing result may be information that includes at least information from which a second estimated motion of the second person P2 can be derived. Details of the second estimated motion will be described later.

[0023] In this embodiment, the sensor 34 includes at least one of a first sensor 34A and a second sensor 34B.

[0024] The first sensor 34A is a sensor 34 mounted on the portable terminal 30 worn on the arm of the second person P2. The first sensor 34A is a sensor 34 mounted on the portable terminal 30, which is, for example, a wristwatch-type wearable device such as a smartwatch. Specifically, the first sensor 34A includes at least one of a position sensor that detects the position of the second person P2 wearing the portable terminal 30, a motion sensor that detects movement, and an attitude sensor that detects attitude (orientation). Examples of the position sensor, motion sensor, and attitude sensor include an IMU (Inertial Measurement Unit) sensor. The IMU sensor is equipped with an angular velocity (gyro) sensor, an acceleration sensor, and a temperature sensor, and is capable of detecting three-dimensional inertial movement.

[0025] The first sensor 34A may further include a sensor for detecting biological information of the second person P2. The biological information may include at least one of information such as heart rate, body temperature, blood oxygen level, blood pressure, and sleep cycle.

[0026] The second sensor 34B is a sensor 34 that detects information about the soles of the second person P2. The information about the soles includes, for example, the pressure distribution on the soles of the second person P2, the temperature distribution on the soles, and the sweat distribution on the soles. An example of the second sensor 34B that measures the pressure distribution on the soles is a sole pressure sensor.

[0027] In this embodiment, a configuration in which the sensor 34 is the first sensor 34A will be described as an example. However, the sensor 34 may be configured to include at least one of the first sensor 34A and the second sensor 34B, and is not limited to a configuration including only the first sensor 34A. When the sensor 34 is configured to include only the first sensor 34A, the portable terminal 30 equipped with the first sensor 34A may be, for example, a portable terminal 30 to be worn on the arm of the person P. When the sensor 34 is configured to include only the second sensor 34B, the portable terminal 30 equipped with the second sensor 34B may be, for example, a portable terminal 30 to be worn on the sole of the leg of the person P.

[0028] The driving unit 35 is a driving mechanism that drives the mobile terminal 30. For example, the driving unit 35 causes the main body of the mobile terminal 30 to vibrate in a vibration pattern according to the control of the processing unit 36.

[0029] The processing unit 36 ​​executes information processing in the mobile terminal 30. The mobile terminal 30 is realized by one or more processors. For example, the processing unit 36 ​​may be realized by having a processor such as a CPU (Central Processing Unit) execute a program, i.e., by software. Furthermore, each of the one or more processing function units included in the processing unit 36 ​​may be realized by a processor such as a dedicated IC (Integrated Circuit), i.e., by hardware. Each of the units may be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or two or more of the units.

[0030] Next, the information processing system 10 will be described.

[0031] The information processing system 10 is one or more information processing devices. The information processing system 10 is configured by one or more dedicated or general-purpose computers.

[0032] The information processing system 10 includes a communication unit 11, an input unit 12, a display unit 13, a storage unit 14, and a processing unit 15. The communication unit 11, the input unit 12, the storage unit 14, and the storage unit 14 and the processing unit 15 are communicatively connected via a bus or the like.

[0033] The communication unit 11 communicates with the imaging device 20 and the mobile terminal 30 via a network NW or the like. The input unit 12 accepts various operations by a user. The display unit 13 outputs various information. The display unit 13 is a display that displays various information, a speaker that outputs various sounds, etc. At least one of the input unit 12 and the display unit 13 may be configured to be provided outside the information processing system 10 and connected to the processing unit 15 so as to be able to communicate with it.

[0034] The storage unit 14 stores various types of data. The storage unit 14 is, for example, a semiconductor memory element such as a RAM or a flash memory, a hard disk, an optical disk, or the like. The storage unit 14 may be a storage device provided outside the information processing system 10. At least a portion of the information stored in the storage unit 14 may be stored in a storage device provided outside the information processing system 10. The storage unit 14 may also be a storage medium that stores or temporarily stores programs and various types of information after downloading them via a local area network (LAN), the Internet, or the like.

[0035] In this embodiment, the storage unit 14 stores information such as photography management information 14A, wearing management information 14B, personal management information 14C, and correspondence management information 14D. Details of these pieces of information will be described later.

[0036] At least one of the display unit 13, the storage unit 14, and an external device such as a server device connected to the information processing system 10 via the network NW functions as an output device 16 that outputs multimodal information, which will be described later.

[0037] The processing unit 15 executes information processing in the information processing system 10 .

[0038] The processing unit 15 includes an acquisition unit 15A, an identification unit 15B, a derivation unit 15C, a comparison unit 15D, an association unit 15E, an output control unit 15F, and an operation control unit 15G.

[0039] The acquisition unit 15A, the identification unit 15B, the derivation unit 15C, the comparison unit 15D, the association unit 15E, the output control unit 15F, and the operation control unit 15G are realized by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU execute a program, i.e., by software. Each of the above units may be realized by a processor such as a dedicated IC, i.e., by hardware. Each of the above units may be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or two or more of the units.

[0040] Furthermore, at least one of the above-mentioned components included in the processing unit 15 may be mounted on an external information processing device such as an external system communicatively connected to the information processing system 10 via a network NW or the like.

[0041] The acquisition unit 15A acquires the captured image and the sensing result information.

[0042] In this embodiment, the image capturing device 20 captures images sequentially in chronological order at a predetermined frame rate and transmits the images sequentially in the order they were captured to the information processing system 10. The sensor 34 of the mobile terminal 30 senses the second person P2 at a predetermined sensing rate in chronological order and transmits the sensing result information sequentially to the information processing system 10.

[0043] The acquisition unit 15A acquires captured images captured by the imaging device 20 and sensing result information from the sensor 34 of the mobile terminal 30. The acquisition unit 15A sequentially accepts captured images from the imaging device 20 and sequentially accepts sensing result information from the mobile terminal 30. The acquisition unit 15A acquires captured images and sensing result information with timestamps having the same imaging timing and sensing timing as captured images and sensing result information with the same acquisition time. Hereinafter, the sensing timing and imaging timing may be referred to as acquisition time.

[0044] The acquisition unit 15A stores the acquired captured image in the photography management information 14A in association with the acquisition time, which is the timing at which the captured image was captured. The acquisition unit 15A also stores the acquired sensing result information in the wearing management information 14B in association with the acquisition time, which is the sensing timing of the sensing result information, and the terminal ID included in the sensing result information.

[0045] 3 is a schematic diagram showing an example of the data structure of the photography management information 14A. The photography management information 14A is a database for managing information related to photographed images photographed by the photography device 20. The data format of the photography management information 14A is not limited to a database.

[0046] For example, the photography management information 14A is a database that associates photographed image ID (Identification), photography device ID, acquisition time, photographed image, first person ID, first analysis result information, and first estimated motion.

[0047] The captured image ID is identification information of the captured image. A captured image ID is assigned to each combination of the camera device 20 that captured the captured image and the acquisition time of the captured image. The camera device ID is identification information of the camera device 20. The acquisition time registered in the photography management information 14A is the timing of the capture of the captured image. The first person ID is identification information of the first person P1 who appears in the corresponding captured image. The first analysis result information is information representing the analysis result obtained by analyzing the captured image of the first person P1 who appears in the corresponding captured image. The first estimated action is an estimated action of the first person P1 who is identified by the corresponding first person ID, estimated by analysis of the corresponding captured image.

[0048] The acquisition unit 15A stores photographed images acquired from the photographing device 20 in the photographing management information 14A in association with the photographing ID of the photographing device 20 that captured the photographed image and the acquisition time, which is the timing at which the photographed image was captured. Furthermore, the acquisition unit 15A assigns a photographed image ID to the photographed image for each photographing device ID and acquisition time, and registers the photographed image in the photographing management information 14A. Therefore, each time a new photographed image is captured by the photographing device 20, the photographed image and the photographed image ID are registered in the photographing management information 14A in association with the photographing device ID of the photographing device 20 that captured the photographed image and the acquisition time, which is the timing at which the photographed image was captured. Furthermore, photographed images captured by the photographing device 20 included in the information management system 1 are sequentially registered in the photographing management information 14A in association with each of multiple different acquisition times in chronological order.

[0049] The first person ID, the first analysis result information, and the first estimated action are derived by the processing unit 15 through processing described later, and are registered in the shooting management information 14A (described in detail later).

[0050] 4 is a schematic diagram of an example of the data configuration of the wearing management information 14B. The wearing management information 14B is a database for managing information related to the portable terminal 30 worn by the second person P2. In other words, the wearing management information 14B is a database for managing information related to the second person P2 wearing the portable terminal 30. The data format of the wearing management information 14B is not limited to a database.

[0051] The wearing management information 14B is a database that associates a terminal ID, a second person ID, an acquisition time, sensing result information, second analysis result information, and a first estimated motion.

[0052] The terminal ID is identification information of the portable terminal 30. The second person ID is identification information of the second person P2 wearing the portable terminal 30 identified by the corresponding terminal ID. In the wearing management information 14B, for each of the multiple second persons P2 managed by the information management system 1, the second person ID of the second person P2 and the terminal ID of the portable terminal 30 worn or to be worn by the second person P2 are registered in advance in association with each other.

[0053] The acquisition time registered in the wearing management information 14B is the sensing timing of the corresponding sensing result information. The sensing result information in the wearing management information 14B is sensing result information by the sensor 34 mounted on the mobile terminal 30 identified by the corresponding terminal ID. The second analysis result information is information representing the analysis result obtained by analyzing the corresponding sensing result information. The second inferred motion is an inferred motion of the second person P2 identified by the corresponding second person ID, which is inferred by analyzing the corresponding sensing result information.

[0054] The acquisition unit 15A stores the sensing result information acquired from the portable terminal 30 in the wearing management information 14B in association with the terminal ID of the portable terminal 30 equipped with the sensor 34 that detected the sensing result information and the acquisition time, which is the sensing timing of the sensing result information. Therefore, each time new sensing result information is acquired by the sensor 34 equipped in the portable terminal 30, the sensing result information is registered in association with the terminal ID of the portable terminal 30 equipped with the sensor 34 that detected the sensing result information, the acquisition time, which is the sensing timing, and the second person ID of the second person P2 wearing the portable terminal 30. Furthermore, the sensing result information detected by the sensor 34 equipped in the portable terminal 30 included in the information management system 1 is sequentially registered in the wearing management information 14B in association with each of a plurality of acquisition times that are different from one another in chronological order.

[0055] The second analysis result information and the second estimated motion are derived by the processing unit 15 through processing described later, and are registered in the wearing management information 14B (described in detail later).

[0056] The storage unit 14 may further store personal management information 14C.

[0057] 5 is a schematic diagram of an example of the data configuration of the personal management information 14C. The personal management information 14C is a database for managing detailed information of the second person P2 who is a target of management by the information management system 1. The data format of the personal management information 14C is not limited to a database.

[0058] The personal management information 14C is a database that associates the second person ID of the second person P2 with the person detailed information of the second person P2 identified by the second person ID. The second person ID is the same as the second person ID registered in the wearing management information 14B. That is, like the wearing management information 14B, the personal management information 14C pre-registers the second person ID of each of the multiple second persons P2 that are to be managed by the information management system 1.

[0059] The person detailed information is information about the second person P2, and preferably includes information that cannot be estimated from the captured image and the sensing result information. For example, the person detailed information includes the name, attribute information, physical information, etc. of the second person P2. The attribute information is information about the attributes of the second person P2, such as the department to which the second person P2 belongs. The physical information is information that represents the physical characteristics of the second person P2. Note that the person detailed information may be any information about the second person P2, and may further include information other than the name, attribute information, and physical information.

[0060] Returning to FIG. 1, the explanation will be continued.

[0061] By adjusting the timestamps, the acquisition unit 15A acquires captured images and sensing result information at the same acquisition time (capture timing, sensing timing) from each of the photographing device 20 and the portable terminal 30. Furthermore, the acquisition unit 15A may store the captured images and sensing result information acquired from each of the photographing device 20 and the portable terminal 30 in the photographing management information 14A and the wearing management information 14B, respectively, and then sequentially acquire the captured images and sensing result information at the same acquisition time (capture timing, sensing timing) from the photographing management information 14A and the wearing management information 14B in chronological order. Note that when acquiring captured images and sensing result information from the photographing management information 14A and the wearing management information 14B, the acquisition unit 15A may sequentially acquire, from among the unprocessed photographed images and sensing result information, the captured images and sensing result information at the same acquisition time that correspond to the acquisition time next (further in the future, more recent) than the acquisition time corresponding to the previously acquired photographed image and sensing result information.

[0062] The identification unit 15B identifies a first person P1 appearing in the photographed image acquired by the acquisition unit 15A. The identification unit 15B performs image analysis of the photographed image using a known method to identify each of one or more persons P appearing in the photographed image as a first person P1. Therefore, when multiple persons P appear in one photographed image, the identification unit 15B identifies the multiple first persons P1 appearing in the one photographed image. The identification unit 15B assigns a first person ID to the identified first person P1, and registers the assigned first person ID in the photography management information 14A in association with the photographed image and the photographed image ID at the acquisition time.

[0063] The identification unit 15B may identify a first person P1 appearing in each of the captured images acquired by the acquisition unit 15A and assign a first person ID to the first person P1 appearing in each of the captured images. In this case, the identification unit 15B assigns a first person ID that can uniquely identify the first person P1 to the first person P1 appearing in each of the multiple captured images captured by at least one of different image capture devices 20 and different capture timings.

[0064] Furthermore, the identification unit 15B may identify the same first person P1 appearing in multiple captured images taken by at least one of different image capture devices 20 and at different capture times using a known image analysis technique, a known tracking technique, etc. In this case, the identification unit 15B assigns the same first person ID to the first person P1 that is determined to be the same person P appearing in multiple captured images taken by at least one of different image capture devices 20 and at different capture times.

[0065] Then, the specification unit 15B registers the assigned first person ID in the photography management information 14A in association with the photographed image and the photographed image ID at the acquisition time.

[0066] The derivation unit 15C derives a first estimated motion of the first person P1 who appears in the captured image acquired by the acquisition unit 15A. The derivation unit 15C also derives a second estimated motion of the second person P2 based on sensing result information by the sensor 34 acquired by the acquisition unit 15A.

[0067] The first estimated motion is an estimated motion of the first person P1 derived from the captured image. The estimated motion includes at least one of an estimated motion of the person P1 and a posture of the person P1.

[0068] The derivation unit 15C derives a first estimated motion of the first person P1 for each first person P1 identified by the identification unit 15B who appears in the photographed images acquired by the acquisition unit 15A. The derivation unit 15C analyzes the photographed images using a known method to derive first analysis result information that is an analysis result of the first person P1 included in the photographed images. The first analysis result information includes, for example, skeletal information that represents the skeleton of the first person P1, facial expression analysis results that represent facial expressions, etc. Then, the derivation unit 15C derives a first estimated motion of the first person P1 based on the first analysis result information.

[0069] 6 is an explanatory diagram illustrating an example of derivation of a first estimated motion by the derivation unit 15C. The derivation unit 15C derives skeletal information of each first person P1 identified by the identification unit 15B and appearing in the captured image 40 from the captured image 40. A known technique may be used to derive the skeletal information. Then, the derivation unit 15C derives a first estimated motion of the first person P1 by estimating the direction of movement, tilt, etc. of the body of the first person P1 represented by the skeletal information based on the skeletal information of the first person P1.

[0070] The derivation unit 15C then registers the first estimated movement estimated for each first person P1, together with the first analysis result information used to derive the first estimated movement, in the photography management information 14A in association with the first person ID of the first person P1 and the captured image ID of the captured image used to derive the first analysis result information. Therefore, as shown in Fig. 3, the photography management information 14A registers, for each captured image ID, the first analysis result information and the first estimated movement of each first person P1 included in the captured image identified by the captured image ID in association with each other.

[0071] Returning to FIG. 1, the explanation will be continued.

[0072] The second estimated motion is an estimated motion of the second person P2 who is wearing the mobile terminal 30, which is derived from the sensing result by the sensor 34.

[0073] The derivation unit 15C derives, based on the sensing result information acquired by the acquisition unit 15A, a second estimated motion of the second person P2 identified by the second person ID associated with the terminal ID included in the sensing result information.

[0074] In particular, when the sensor 34 mounted on the mobile terminal 30 is the first sensor 34A, the derivation unit 15C derives, as second analysis result information, information such as posture, behavior, relative direction, position, center of gravity, and walking habits using a known method based on the angular velocity, acceleration, and other detection results of the IMU sensor included in the sensing result information. The derivation unit 15C also derives, as second analysis result information, biological information such as heart rate, body temperature, blood oxygen concentration, blood pressure, and sleep cycle included in the sensing result information. The derivation unit 15C then derives, using a known method, a second estimated motion of the second person P2 from the derived posture, relative direction, position, and other information.

[0075] Furthermore, when the sensor 34 mounted on the mobile terminal 30 is the second sensor 34B, the derivation unit 15C derives information included in the sensing result information, such as the pressure distribution on the soles of the second person P2, the temperature distribution on the soles of the feet, and the sweat distribution on the soles of the feet, as second analysis result information. Then, the derivation unit 15C derives a second estimated movement of the second person P2 from the derived second analysis result information, such as the pressure distribution on the soles of the feet, by a known method.

[0076] Then, the derivation unit 15C associates the second inferred motion derived for each second person P2 with the sensing result information and second analysis result information used to derive the second inferred motion, and registers the second inferred motion in the wearing management information 14B. Therefore, as shown in Fig. 4 , the sensing result information, second analysis result information, and second inferred motion of the second person P2 are registered in association with each other for each second person ID of the second person P2 in the wearing management information 14B.

[0077] The derivation unit 15C may derive the first estimated motion for all first persons P1 identified by the identification unit 15B and included in the captured image acquired by the acquisition unit 15A, for which no second information is associated with the first information about the first persons P1. Furthermore, if the second information is associated with the first information about all first persons P1 identified by the identification unit 15B and included in the captured image acquired by the acquisition unit 15A, the derivation unit 15C may omit the processing by the comparison unit 15D and the association unit 15E, which will be described later. Details of the first information and the second information will be described later.

[0078] Next, the comparison unit 15D will be described.

[0079] The comparison unit 15D compares the first estimated motion of the first person P1 appearing in the captured image with the second estimated motion of the second person P2 derived based on the sensing result information for each captured image at the same acquisition time, and then determines whether the first estimated motion and the second estimated motion match.

[0080] Assume a scene in which multiple first persons P1 appear in a captured image acquired at a certain capture time, and a first inferred motion is derived for each of the multiple first persons P1. In this case, the comparison unit 15D compares the first inferred motion of each of the multiple first persons P1 with the second inferred motion of each of the multiple second persons P2 that is derived from the sensing result information of each of the multiple second persons P2 that are managed by the information management system 1.

[0081] For example, as shown in FIGS. 3 and 4 , a scene is assumed in which first inferred movements of first person IDs "A," "B," and "C" are derived from captured images with captured image IDs "0001" and "0011" acquired at a certain acquisition time "xx, yy, zz01." Also, a scene is assumed in which second inferred movements of second person IDs "1," "2," and "3" are derived as sensing result information acquired at the acquisition time "xx, yy, zz01" by the sensors 34 of multiple second persons P2 managed by the information management system 1. In this case, the comparison unit 15D determines that the first inferred movement "walking" and the second inferred movement "walking" are the same inferred movement. Furthermore, the comparison unit 15D determines that the first inferred movement "squatting" and the second inferred movement "bending right hand inward and lowering body position" are the same inferred movement.

[0082] When a new first person P1 who does not appear in any of the previously acquired images by the acquisition unit 15A is identified from the currently acquired captured image, the comparison unit 15D may compare the first estimated motion of the new first person P1 with the second information about the second person P2, which is not associated with the first information. The association will be described in detail later.

[0083] Returning to FIG. 1, the explanation will be continued.

[0084] If the first estimated action and the second estimated action match based on the comparison result between the first estimated action and the second estimated action by the comparison unit 15D, the matching unit 15E matches the first information about the first person P1 appearing in the captured image with the second information obtained from the mobile terminal 30.

[0085] Specifically, the association unit 15E associates the first information about the first person P1, whose first and second estimated actions match, with the second information acquired from the portable terminal 30. The match between the first and second estimated actions indicates that the degree of match between the first and second estimated actions is equal to or exceeds a threshold at which the first and second estimated actions are considered identical. Therefore, the match between the first and second estimated actions is not limited to a complete match. The threshold may be changeable as needed by a user operating the input unit 12, for example.

[0086] 3 and 4, it is assumed that the comparison unit 15D determines that the first inferred action “walking” and the second inferred action “walking” are the same inferred action based on the captured image and sensing result information acquired at a certain acquisition time “xx, yy, zz01.” In this case, the association unit 15E associates the first information of a first person P1 identified by a first person ID “C” corresponding to the first inferred action “walking” with the second information of a second person P2 identified by a second person ID “1” corresponding to the second inferred action “walking,” which have been determined to be the same inferred action at the same acquisition time.

[0087] The first information is information related to a first person P1. Specifically, the first information includes first analysis result information of the first person P1 appearing in the captured image. The first information may further include a first estimated motion derived from the first analysis result information. The first information may further include a captured image in which the first person P1 appears.

[0088] 3, at least a part of the photographed image ID, photographing device ID, acquisition time, photographed image, first analysis result information, and first estimated motion associated with the first person ID in the photographing management information 14A is treated as first information about the first person P1 identified by the first person ID. Also, as shown in FIG. 3, the photographing management information 14A registers first information for each of a plurality of photographed images with different acquisition times and photographing devices 20.

[0089] That is, the first information of the first person P1 includes information about a plurality of captured images of the first person P1 that differ in at least one of the acquisition time and the image capture device 20. Specifically, for example, the first information of the first person ID "C" includes the captured image, first analysis result information, and first estimated action for each captured image ID, image capture device ID, and acquisition time that are associated with the first person ID "C" in the image capture management information 14A. In other words, the first information of the first person ID "C" includes information about the first person P1 identified by the first person ID "C" that is derived from currently and previously captured images of the first person P1.

[0090] The second information includes information acquired from the portable device 30 worn by the second person 2P who is the wearer. In other words, the second information is information about the second person P2 who is the wearer of the portable device 30. In detail, the second information includes second analysis result information of the second person P2 who is the wearer of the portable device 30 equipped with the sensor 34 that detected the sensing result information. The second analysis result information includes data received from the sensor 34. The second analysis result information may also include an analysis result of the sensing result by the sensor 34. The second information may also include a second estimated motion derived from the second analysis result information. The second information may also include sensing result information of the second person P2. The second information may also include person detail information of the second person P2.

[0091] 4 and 5, the device ID, acquisition time, sensing result information, second analysis result information, second estimated motion, and person detailed information associated with the second person ID are treated as second information related to the second person P2 identified by the second person ID. Also, as shown in FIG. 4, the wearing management information 14B registers second information for each of the plurality of pieces of sensing result information acquired at different times. Therefore, the second information of the second person P2 includes information related to the plurality of pieces of sensing result information of the second person P2 acquired at different times.

[0092] That is, the second information of the second person P2 includes the device ID of the second person P2, as well as sensing result information, second analysis result information, and second estimated motion corresponding to each of a plurality of different acquisition times. Specifically, for example, the second information of the second person ID "1" includes the sensing result information, second analysis result information, and second estimated motion for each acquisition time associated with the second person ID "1" in the wearing management information 14B. Therefore, the second information of the second person ID "1" includes information related to current and past sensing result information regarding the second person P2 identified by the second person ID "1." Furthermore, the second information of the second person ID "1" includes detailed person information associated with the second person ID "1" in the personal management information 14C.

[0093] Returning to FIG. 1, the explanation will be continued.

[0094] The matching unit 15E matches the first information about the first person P1 and the second information about the second person P2 whose estimated actions match by matching the first person ID of the first person P1 and the second person ID of the second person P2 whose estimated actions match and registering them in the matching management information 14D.

[0095] 7 is a schematic diagram of an example of the data configuration of the association management information 14D. The association management information 14D is a database in which first person IDs and second person IDs are associated with each other. The data format of the association management information 14D is not limited to a database.

[0096] The association unit 15E associates the first person ID of the first person P1 and the second person ID of the second person P2, whose first and second inferred actions match, and registers them in the association management information 14D. Through this registration process, the first information corresponding to the first person ID registered in association with the association management information 14D and the second information corresponding to the second person ID associated with the first person ID in the association management information 14D are associated with each other.

[0097] Furthermore, by registering in correspondence management information 14D a first person ID and a second person ID for which it has been determined that the first estimated action and the second estimated action match, first information and second information regarding the captured image and sensing result information including the latest acquisition time and past acquisition time are associated with each other for a first person P1 identified by the first person ID and a second person P2 identified by the second person ID.

[0098] 7, the association management information 14D may be a database in which first information corresponding to a first person ID and second information corresponding to a second person ID are further associated with each other. In this case, the association unit 15E may update the first information and second information corresponding to the first person ID and the second person ID in the association management information 14D over time every time new first information and new second information are derived by the acquisition unit 15A, the identification unit 15B, and the derivation unit 15C.

[0099] Returning to FIG. 1, the explanation will be continued.

[0100] The output control unit 15F outputs multimodal information in which the first information and the second information are associated with each other based on the comparison result between the first estimated action and the second estimated action to the output device 16. As described above, the display unit 13, the storage unit 14, and an external device such as a server device connected to the information processing system 10 via the network NW function as the output device 16. Therefore, the output control unit 15F outputs the multimodal information to at least one of the display unit 13, the storage unit 14, and the external device.

[0101] The output control unit 15F uses, for example, the association management information 14D as the multimodal information. In this case, the output control unit 15F outputs the association management information 14D, which associates a first person ID, the first information, a second person ID, and the second information, to the output device 16 as multimodal information. As described above, the first information of the first person P1 includes information on a plurality of captured images of the first person P1 that are acquired at different times and / or by different image capture devices 20. The second information of the second person P2 includes information on a plurality of pieces of sensing result information of the second person P2 that are acquired at different times. The second information of the second person P2 may also include detailed person information about the second person P2.

[0102] Therefore, the multimodal information includes, for each person P determined to have the same estimated motion, multimodal information that associates information on one or more captured images captured by one or more image capturing devices 20 (first information), information on multiple captured images captured at different acquisition times (first information), information on sensing result information by the sensor 34 (second information), and person detailed information (second information), etc. Multimodal information is information that combines different types of information.

[0103] Therefore, the output control unit 15F can output the multimodal information for each person P by outputting the multimodal information to the output device 16.

[0104] In addition, the processing unit 15 may send operation instruction information to the mobile terminal 30 to prompt an operation, and may associate the first information with the second information based on a comparison result of the first estimated operation and the second estimated operation derived based on each of the captured image and sensing result information acquired by the acquisition unit 15A after the operation instruction information is sent.

[0105] In detail, the processing unit 15 repeatedly executes the above process each time the acquisition unit 15A acquires a newly captured image and sensing result information detected at the same sensing timing as the capture timing of the captured image. However, when the processing unit 15 executes the process for a predetermined period, there may be a second person P2 whose second information has not yet been associated with the first information. In other words, even if the processing unit 15 repeatedly executes the above process each time it acquires a captured image and sensing result information acquired at the same acquisition time, there may be a second person P2 whose second information has not yet been associated with the first information among the second persons P2 managed by the information management system 1.

[0106] The operation control unit 15G determines whether a predetermined period of time has elapsed since the start of processing by the processing unit 15. The predetermined period of time may be determined in advance. The predetermined period of time may also be changeable as appropriate by an operation instruction from the input unit 12 by a user or the like.

[0107] When the operation control unit 15G determines that a predetermined period of time has elapsed, it determines whether or not there is a second person ID registered in the personal management information 14C that is not associated with a first person ID in the association management information 14D. Based on this determination, the operation control unit 15G determines whether or not there is a second person P2 of the second information that is not associated with the first information.

[0108] When the operation control unit 15G determines that there is a second person P2 of the second information that is not associated with the first information, the operation control unit 15G transmits operation instruction information to the portable terminal 30 of the second person P2 that is not associated with the first information. Note that the derivation unit 15C may transmit the operation instruction information to all portable terminals 30. However, it is preferable that the derivation unit 15C transmits the operation instruction information to the portable terminal 30 of the second person P2 that is not associated with the first information.

[0109] The action instruction information is information for prompting the second person P2 to perform some action.

[0110] When the portable terminal 30 worn by the second person P2 vibrates, it is presumed that the second person P2 will perform an action to visually recognize the portable terminal 30 worn by the second person P2. Therefore, for example, the operation instruction information may be information for instructing the portable terminal 30 to vibrate with a specific vibration pattern. In this case, the processing unit 36 ​​of the portable terminal 30 that has received the operation instruction information controls the driving unit 35 to vibrate with the vibration pattern represented by the operation instruction information. Therefore, the portable terminal 30 vibrates with that vibration pattern.

[0111] Furthermore, the action instruction information may be information for displaying instruction information instructing the second person P2 to perform some action on the UI unit 32 of the mobile terminal 30. In this case, the processing unit 36 ​​of the mobile terminal 30 that has received the action instruction information displays the instruction information instructing the second person P2 to perform the action on the UI unit 32.

[0112] In this embodiment, the description will be made assuming that the operation instruction information is information for instructing the mobile terminal 30 to vibrate with a specific vibration pattern. Note that, when there are multiple mobile terminals 30 to which the operation instruction information is to be sent, it is preferable that the operation control unit 15G sequentially sends the operation instruction information to the multiple mobile terminals 30 at predetermined intervals. This control enables the operation control unit 15G to vibrate the multiple mobile terminals 30 individually and sequentially with a time lag.

[0113] The comparison unit 15D of the processing unit 15 compares the first estimated operation and the second estimated operation derived based on the captured image and the sensing result information acquired after the operation instruction information is transmitted, and the correspondence unit 15E corresponds the first information and the second information based on the comparison result in the same manner as described above.

[0114] By transmitting the operation instruction information to the portable terminal 30, the second person P2 wearing the portable terminal 30 is likely to perform some operation predicted from the operation instruction information. Therefore, by associating the first information with the second information based on a comparison result between the first inferred operation and the second inferred operation derived based on the captured image and the sensing result information acquired after the transmission of the operation instruction information, it is possible to efficiently associate the first information with the second information that has not been associated.

[0115] Next, an example of the flow of information processing executed by the information processing system 10 of this embodiment will be described.

[0116] 8 is a flowchart showing an example of the flow of information processing executed by the information processing system 10 of this embodiment. The processing unit 15 of the information processing system 10 sequentially acquires captured images and sensing result information at the same acquisition time from the imaging device 20 and the mobile terminal 30 in chronological order, and executes the processing shown in FIG. 8 each time it acquires a captured image and sensing result information at a new acquisition time.

[0117] The acquisition unit 15A acquires captured images and sensing result information acquired at the same acquisition time from the imaging device 20 and the mobile terminal 30, respectively (step S100). The acquisition unit 15A stores the acquired captured images and sensing result information in the imaging management information 14A and the wearing management information 14B, respectively (step S102). Specifically, the acquisition unit 15A stores the acquired captured images in the imaging management information 14A in association with the acquisition time, which is the timing at which the captured images were captured, the imaging device ID, and the captured image ID. The acquisition unit 15A also stores the acquired sensing result information in the wearing management information 14B in association with the acquisition time, which is the sensing timing of the sensing result information, and the terminal ID included in the sensing result information.

[0118] The identification unit 15B identifies the first person P1 appearing in the photographed image acquired in step S100 (step S104). If multiple first persons P1 appear in the photographed image, the identification unit 15B identifies the multiple first persons P1. Furthermore, if photographed images captured at the same acquisition time are acquired from multiple image capture devices 20 in step S100, the identification unit 15B identifies the first person P1 included in each of the multiple photographed images.

[0119] The identification unit 15B assigns a first person ID to the first person P1 identified in step S104 (step S106). For example, assume that the identification unit 15B identifies the same first person P1 appearing in multiple captured images taken at the same acquisition time in step S100 using a known image analysis technique, a known tracking technique, or the like. In this case, the identification unit 15B assigns the same first person ID to the first person P1 that is determined to be the same person appearing in multiple captured images taken by different image capture devices 20 at the same acquisition time. The identification unit 15B then registers the assigned first person ID in the photography management information 14A in association with the captured image and the captured image ID at the acquisition time.

[0120] The derivation unit 15C determines whether the second information of the second person P2 is associated with the first information of all the first persons P1 appearing in the photographed images identified in step S104 (step S108). The derivation unit 15C makes the determination in step S108 by determining whether the first person IDs of all the first persons P1 appearing in the photographed images identified in step S104 are registered in the association management information 14D and whether second person IDs are registered in the association management information 14D in association with all the first person IDs.

[0121] If the determination in step S108 is affirmative (step S108: Yes), the process proceeds to step S126. If the determination in step S108 is negative (step S108: No), the process proceeds to step S110.

[0122] In step S110, the derivation unit 15C derives a first estimated motion of the first person P1 identified in step S104 who appears in the captured image acquired in step S100 (step S112). Furthermore, the derivation unit 15C derives a second estimated motion of the second person P2 identified by the second person ID included in the sensing result information based on the sensing result information acquired by the sensor 34 in step S100 (step S112).

[0123] The comparison unit 15D compares the first estimated motion derived for each first person P1 in step S110 with the second estimated motion derived for each second person P2 in step S112, and determines whether there are any first person P1 and second person P2 for which the first estimated motion and the second estimated motion match (step S114). If the determination in step S114 is negative (step S114: No), the process proceeds to step S120. If the determination in step S114 is positive (step S114: Yes), the process proceeds to step S116.

[0124] In step S116, the association unit 15E associates the first information about the first person P1, whose first inferred action and the second inferred action match, with the second information about the second person P2, based on the comparison result of step S114 between the first inferred action and the second inferred action, and registers the associated information in the association management information 14D (step S116). In detail, the association unit 15E associates the first information about the first person P1, whose first inferred action and the second inferred action match, with the second information about the second person P2, by registering the associated information in the association management information 14D.

[0125] Next, the output control unit 15F outputs the multimodal information in which the first information and the second information are associated with each other to the output device 16 (step S118). The output control unit 15F uses, for example, the association management information 14D as the multimodal information. In this case, the output control unit 15F outputs the association management information 14D in which the first person ID, the first information, the second person ID, and the second information are associated with each other to the output device 16 as the multimodal information.

[0126] As described above, the first information of the first person P1 includes information on a plurality of captured images of the first person P1 that differ in at least one of the acquisition time and the image capture device 20. The second information of the second person P2 includes information on a plurality of pieces of sensing result information of the second person P2 that differ in acquisition time. The second information of the second person P2 may also include detailed person information of the second person P2. Therefore, the output control unit 15F can output multimodal information for each person P by outputting the multimodal information to the output device 16.

[0127] Next, the operation control unit 15G determines whether a predetermined period of time has elapsed since the processing unit 15 started the series of processes from step S100 to step S126 (step S120). If the determination in step S120 is negative (step S120: No), the process proceeds to step S126. If the determination in step S120 is positive (step S120: Yes), the process proceeds to step S122.

[0128] In step S122, the operation control unit 15G determines whether there is a second person P2 of the second information that is not associated with the first information (step S122). The operation control unit 15G executes the determination of step S122 by determining whether there is a second person ID that is not associated with a first person ID and registered in the association management information 14D among the multiple second person IDs registered in the personal management information 14C, i.e., the second person IDs of all second persons P2 that are managed by the information management system 1.

[0129] If the determination in step S122 is affirmative (step S122: Yes), the process proceeds to step S124. In step S124, the operation control unit 15G transmits operation instruction information to the portable terminal 30 identified by the terminal ID associated with the second person ID not associated with the first information (step S124). Then, the processing unit 15 sets the acquisition time of the captured image and sensing result information to be acquired next by the acquisition unit 15A to the time following the previous acquisition time, and then returns to step S100. The next time may be any time later (future) than the previous acquisition time.

[0130] On the other hand, if the determination in step S122 is negative (step S122: No), the processing unit 15 proceeds to step S126. In step S126, the processing unit 15 determines whether to end the processing (step S126). For example, the processing unit 15 executes the determination in step S126 by determining whether an instruction signal indicating the end of the processing has been received from the input unit 12. If the determination in step S126 is negative (step S126: No), the processing unit 15 sets the acquisition time of the captured image and sensing result information to be acquired next by the acquisition unit 15A to the time following the previous acquisition time, and then returns to step S100. If the determination in step S126 is positive (step S126: Yes), the processing unit 15 ends this routine.

[0131] As described above, the information processing system 10 of this embodiment includes at least one processor. When a first estimated motion of a first person P1 appearing in a captured image matches a second estimated motion of a wearer of the portable device 30 (a second person P2) derived based on a sensing result by the sensor 34 mounted on the portable device 30, the processor (processing unit 15) outputs, to the output device 16, multimodal information that associates first information about the first person P1 in the captured image with second information acquired from the portable device 30.

[0132] The first information of the first person P1 includes information about a captured image in which the first person P1 appears. The second information includes information acquired from the portable terminal 30 worn by the wearer. Therefore, the multimodal information includes, for each person P, information that associates first information about a captured image in which the person P appears and second information about the portable terminal 30 worn by the person P based on the comparison result between the first estimated motion and the second estimated motion. Therefore, the information processing system 10 of the present embodiment can integrate and manage multimodal information in which multiple types of information about the person P are associated with each person P.

[0133] Therefore, the information processing system 10 of this embodiment can manage multiple types of information related to the person P in an integrated manner.

[0134] Furthermore, the multimodal information for each person P generated by the information processing system 10 of this embodiment is suitably applied to various processes that utilize multimodal information.

[0135] Furthermore, as described above, at least one processor of the information processing system 10 of this embodiment associates first information about the first person P1 whose first estimated action and second estimated action match with second information obtained from the mobile terminal 30 worn by the wearer (second person P2).

[0136] Therefore, the information processing system 10 of the present embodiment can identify a first person P1 and a wearer of the portable terminal 30 (a second person P2) whose first estimated motion and second estimated motion derived by different methods match as the same person P, and can associate the first information of the first person P1 with the second information of the second person P2. Therefore, compared to a method of recognizing a person only from information detected by a beacon or the like, or conventional technology of recognizing a person only from a captured image, the information processing system 10 of the present embodiment can associate the first information with the second information with a high degree of accuracy even in an environment where it is difficult to distinguish between each person P in real space.

[0137] The first information includes first analysis result information of a first person P1 who appears in the captured image, and the second information includes second analysis result information of data from the sensor 34.

[0138] Therefore, by the information processing system 10 associating the first information with the second information for each person P, it becomes possible to integrate and manage information relating to the captured image and the sensing result information.

[0139] In addition, at least one processor of the information processing system 10 of this embodiment transmits operation instruction information to the mobile terminal 30 to prompt an operation, and matches the first information with the second information based on the comparison result of the first estimated operation and the second estimated operation derived based on each of the captured image and sensing result information acquired after the transmission of the operation instruction information.

[0140] Therefore, in the information processing system 10 of this embodiment, even if the real space is an environment in which it is difficult to distinguish between the movements of multiple people P, the first information and the second information can be appropriately associated based on the comparison results between the first estimated movement and the second estimated movement.

[0141] Furthermore, when at least one processor of the information processing system 10 of this embodiment executes processing for a predetermined period, if there is a second person P2 of second information that is not associated with the first information, it transmits operation instruction information to the mobile terminal 30 attached to the unassociated second person P2.

[0142] In this way, by sending operation instruction information when certain conditions are met, the information processing system 10 of this embodiment can efficiently associate the first information with the second information by sending the minimum necessary operation instruction information.

[0143] The sensor 34 includes at least one of a first sensor 34A mounted on the mobile terminal 30 worn on the arm of the second person P2 and a second sensor 34B that detects information relating to the soles of the legs of the second person P2.

[0144] By using various types of sensors 34, the information processing system 10 of this embodiment can manage even more multimodal information.

[0145] Furthermore, at least one processor of the information processing system 10 of this embodiment derives a first estimated motion of the first person P1 based on skeletal information of the first person P1 appearing in the captured image.

[0146] Therefore, the information processing system 10 of the present embodiment can derive a more detailed first estimated action. Therefore, by using the first estimated action, the information processing system 10 of the present embodiment can associate the first information with the second information in a shorter time, even when a larger number of first persons P1 appear in the captured image.

[0147] In addition, when at least one processor of the information processing system 10 of this embodiment identifies a new first person P1 that does not appear in any previously acquired captured image, it compares the first estimated movement of the first person P1 with the second estimated movement of the second person P2 that is not associated with the first information regarding the second person P2.

[0148] Therefore, the information processing system 10 of the present embodiment can narrow down the second inferred movements of the second person P2 referred to during the comparison process to the second inferred movements of the second person P2 identified by the second person ID that is not associated with the first information, thereby enabling the information processing system 10 to efficiently execute the comparison process.

[0149] In addition, at least one processor of the information processing system 10 of this embodiment continues to derive the first estimated motion of one or more first persons P1 that appear in the captured image, for which second information is not associated with first information regarding the first person P1.

[0150] Therefore, the information processing system 10 of this embodiment can omit the comparison process between the first estimated movement of the first person P1 and the second estimated movement of the second person P2 for the first person P1 who has already been matched, thereby making it possible to improve processing efficiency.

[0151] In addition, at least one processor of the information processing system 10 of this embodiment identifies the same first person P1 that appears in multiple captured images taken by at least one of different imaging devices 20 and different timings, and matches the first information about the identified first person P1 with the second information about the second person.

[0152] For this reason, in the information processing system 10 of this embodiment, for a first person P1 of first information that has once been associated with the second information of a second person P2, the first information remains associated with the second information of the second person P2 even if at least one of the acquisition time and the image capture device 20 is different. Therefore, in the information processing system 10 of this embodiment, for a first person P1 and a second person P2 that have once been associated, it is possible to manage the first information and the second information for the first person P1 and the second person P2 as multimodal information not only at the time when the association was made, but also over all periods in the past and future from that time.

[0153] Furthermore, in the information processing system 10 of this embodiment, by executing the above processing, even if a person P moves in real space from the shooting angle of one photographing device 20 to the shooting angle of another photographing device 20, it is possible to associate the first information of a first person P1 who appears in a photographed image of the person P with the second information of a second person P2 who is the person P.

[0154] The following description will be made with reference to Figure 2 and Figures 9A to 9C. Figures 9A to 9C are explanatory diagrams showing a series of usage scenes of the information processing system 10.

[0155] 2, assume that the information management system 1 includes multiple image capture devices 20A to 20C. The image capture angle and image capture direction of each of the multiple image capture devices 20 are adjusted in advance so that at least some of the image capture angles do not overlap. Also assume that five people P who are subject to management by the information management system 1 are present in a real space that includes the image capture range of the image capture device 20, and each of the five people P is wearing a mobile terminal 30.

[0156] 9A is a schematic diagram showing an example of captured images 50 (captured images 50A to 50C) captured by each of the image capturing devices 20A to 20C at a first acquisition time of the real space in the state shown in FIG. 2, and wearing management information 14B including sensing result information 60 at the same first acquisition time (sensing timing) as the captured images 50. The first acquisition time is an example of an acquisition time.

[0157] When the photographed images 50 shown in FIG. 9A are acquired, the identification unit 15B identifies the first person P1 appearing in each of the photographed images 50 (photographed images 50A to 50C) and assigns a first person ID to the first person P1 appearing in each of the photographed images 50. FIG. 9A shows a scene in which the identification unit 15B assigns first person IDs "A" and "B" to the two first persons P1 appearing in the photographed image 50A, respectively. FIG. 9A also shows a scene in which the identification unit 15B assigns first person IDs "C" and "D" to the two first persons P1 appearing in the photographed image 50B, respectively. FIG. 9A also shows a scene in which the identification unit 15B assigns first person ID "E" to the single first person P1 appearing in the photographed image 50C.

[0158] The derivation unit 15C then derives a first estimated motion of each of the first persons P1 appearing in the captured image 50. The derivation unit 15C also derives a second estimated motion of each of the second persons P2 identified by the second person IDs "1" to "5" managed by the information management system 1, based on the sensing result information 60. The comparison unit 15D compares the derived first estimated motion with the second estimated motion. The association unit 15E associates first information about the first person P1 whose first estimated motion and second estimated motion match with second information about the sensing result information of the second person P2.

[0159] 9B is an explanatory diagram showing an example of the relationship between the first person P1 of the first information and the second person P2 of the second information, which are associated based on the captured image 50 at the first acquisition time and the sensing result information 60. It is assumed that, as a result of the above processing by the processing unit 15, the first information of the first person ID "A" is associated with the second information of the second person ID "2," the first information of the first person ID "D" is associated with the second information of the second person ID "1," and the first information of the first person ID "E" is associated with the second information of the second person ID "3" at the first acquisition time. It is also assumed that, at the first acquisition time, there is no second person P2 with a matching estimated motion for the first person IDs "C" and "B," and therefore the first information is not associated with the second information.

[0160] In this case, the multimodal information is managed in association with the person P identified by the first person ID "A" and the second person ID "2," the person P identified by the first person ID "D" and the second person ID "1," and the person P identified by the first person ID "E" and the second person ID "3," who are associated with the first information and the second information. On the other hand, at this first acquisition time, the multimodal information is not managed in association with the person P identified by the first person IDs "C" and "B."

[0161] It is assumed that, from this state, toward the second acquisition time, which is the acquisition time following the first acquisition time, the first person P1, who was assigned the first person ID "C" at the first acquisition time, moves from the angle of view of photographed image 50B to the angle of view of photographed image 50C. It is also assumed that the first person P1, who was assigned the first person ID "A" at the first acquisition time, moves from the angle of view of photographed image 50A to the angle of view of photographed image 50B.

[0162] As described above, the identification unit 15B identifies the same first person P1 appearing in multiple captured images taken by at least one of different image capture devices 20 and at different capture times, using a known image analysis technique, a known tracking technique, etc. That is, the identification unit 15B assigns the same first person ID to the first person P1 that is determined to be the same person P appearing in multiple captured images taken by at least one of different image capture devices 20 and at different capture times.

[0163] Therefore, even if at least one of the acquisition time and the captured image 50 in which the person P appears is different, each of the first person IDs "A" to "E" assigned at the first acquisition time is assigned to the first person P1 who is determined to be the same person P as the first person P1 identified by the first person ID. In other words, the first person P1 assigned the first person ID "A" who moves from the angle of view of the captured image 50A to the captured image 50B can also continue to be tracked within the captured image 50B.

[0164] As described above, the derivation unit 15C may derive the first inferred motion for each first person P1 whose first information is not associated with the second information among all first persons P1 identified by the identification unit 15B and included in the captured image acquired by the acquisition unit 15A. Therefore, the processing unit 15 may omit the derivation, comparison process, and association process of the first inferred motion and the second inferred motion for the first person IDs "A," "D," and "E" who are first persons P1 whose first information is associated with the second information at the first acquisition time that is earlier than the second acquisition time.

[0165] Then, by repeatedly executing the above process each time the processing unit 15 acquires a captured image and sensing result information at a new acquisition time, the first person P1 of the first information that is not associated with the second information at the second acquisition time will be associated with the second information at one of the future acquisition times. Also, for example, the first information of the first person ID "B" that was not associated with the second information at the first acquisition time will be associated with the second information of the second person ID "4" at the second acquisition time. Also, by repeating this process, the first information of the first person P1 assigned the first person ID "C" may also be associated with the second information at a future time.

[0166] Furthermore, as described above, each of the photography management information 14A and the wearing management information 14B has registered therein a plurality of pieces of first information and second information acquired at different times. Therefore, by associating the first information and the second information with a certain acquisition time, all of the first information and the second information acquired at and before the acquisition time are associated with the person P, who is the first person P1 of the associated first information and the second person P2 of the associated second information. Therefore, the information processing system 10 can manage multimodal information in which the first information and the second information are associated for each person P, from the past acquisition time to the most recent acquisition time.

[0167] Furthermore, when the identification unit 15B identifies a new first person P1 who does not appear in any previously acquired captured image 50, the identification unit 15B assigns a new first person ID to the first person P1. Then, the comparison unit 15D compares the first estimated motion of the first person P1 with the second estimated motion of a second person P2 who is not associated with the first information of the second person P2 among the second persons P2 managed by the information management system 1.

[0168] In this way, the processing unit 15 associates the first information and the second information of the newly identified first person P1 based on the comparison result with the second estimated movement of the second person P2 that is not associated with the first information. This allows the information processing system 10 to narrow down the second estimated movements to be compared, thereby improving processing efficiency.

[0169] The identification unit 15B may assign a first person ID capable of uniquely identifying the first person P1 to a first person P1 appearing in each of a plurality of captured images 50 captured by at least one of different image capture devices 20 and different capture timings. In this case, for example, a different first person ID is assigned to a first person P1 appearing in a captured image 50 captured by a certain mobile terminal 30 at a certain capture time and a first person P1 who is the same person P as the first person P1 appearing in a captured image 50 captured at a different capture time or by a different mobile terminal 30. However, by the processing unit 15 performing the above-described derivation process, comparison process, and association process of the first inferred action and the second inferred action, for the same person P, the second information of the second person ID of the same second person P2 is associated with the first information of a plurality of different first person IDs. Therefore, even if the identification unit 15B assigns a new first person ID to each of multiple captured images 50 captured by at least one of different imaging devices 20 and different imaging timings, it is possible to manage multimodal information that associates first information and second information for each person P.

[0170] In the above embodiment, the sensor 34 is the first sensor 34A as an example. However, the sensor 34 mounted on the mobile terminal 30 may be configured to include at least one of the first sensor 34A and the second sensor 34B. For example, the sensor 34 may be the second sensor 34B. The sensor 34 may also be configured to include the first sensor 34A and the second sensor 34B. The sensor 34 may also be a sensor that can sense a sensing result from which the second estimated movement of the second person P2 can be derived, and may be configured to include a type of sensor 34 other than the first sensor 34A and the second sensor 34B.

[0171] In the above embodiment, the comparison unit 15D compares the first estimated motion with the second estimated motion for each acquisition time. However, the processing unit 15 may identify a group of first estimated motions at the latest acquisition time and a group of previously acquired times for each of the multiple first persons P1. The processing unit 15 may also identify a group of second estimated motions at the latest acquisition time and a group of previously acquired times for each of the multiple second persons P2. The processing unit 15 may then associate the first information and the second information for which the groups of estimated motions match based on a comparison result obtained by comparing the group of first estimated motions for each of the multiple first persons P1 with the group of second estimated motions for each of the multiple second persons P2.

[0172] Second Embodiment In this embodiment, a form will be described in which first information and second information are associated based on a captured image in which a second person P2, who is a person P wearing a mobile terminal 30, appears as a first person P1.

[0173] In this embodiment, parts having the same functions or configurations as those in the above embodiment are given the same reference numerals, and detailed description thereof will be omitted.

[0174] FIG. 10 is an explanatory diagram of an example of an information management system 1B according to this embodiment.

[0175] The information management system 1B is a system for managing multiple types of information related to a person P in an integrated manner, similar to the information management system 1.

[0176] The information management system 1B includes an information processing system 10B, one or more image capturing devices 20, one or more mobile terminals 30B, and one or more communication devices 70. The information processing system 10B, the image capturing devices 20, the mobile terminals 30B, and the communication devices 70 are communicatively connected via a network NW or the like.

[0177] The photographing device 20 is the same as that in the above embodiment.

[0178] The communication device 70 is a device that transmits a wireless signal at predetermined intervals. The communication device 70 is equipped with a function for communicating via Bluetooth (registered trademark), for example, and transmits and receives wireless signals based on a wireless communication standard such as a short-range wireless communication standard (Bluetooth Low Energy: BLE). Note that the wireless communication standard used by the communication device 70 is not limited to one that uses Bluetooth. For example, the communication device 70 may perform wireless communication using an infrared communication standard such as IrDA (Infrared Data Association).

[0179] The wireless signal transmitted from the communication device 70 includes a communication device ID (Identification), which is identification information of the communication device 70 that is the source of the wireless signal. The communication device 70 is fixedly placed in real space.

[0180] The image capturing device 20 is the same as in the above embodiment. The portable terminal 30B is a portable terminal worn by the second person P2, similar to the portable terminal 30 in the above embodiment.

[0181] FIG. 11 is an explanatory diagram of an example of an application form of the information management system 1B of this embodiment.

[0182] In this embodiment, an example will be described in which the information management system 1B is applied to a real space RS such as a commercial facility where a wide variety of people P may enter and exit.

[0183] 11 , a plurality of image capturing devices 20 and a plurality of communication devices 70 are arranged in a real space RS such as a commercial facility. As described in the above embodiment, the plurality of image capturing devices 20 are arranged so as to be able to capture an image of a predetermined space in the real space RS. In this embodiment, the description will be given assuming that the predetermined space is, for example, a commercial facility. Note that the predetermined space is not limited to this space.

[0184] 11 shows an example in which the information management system 1 includes a plurality of image capture devices 20A to 20C. The image capture angles and image capture directions of the image capture devices 20 are adjusted in advance so that at least some of the image capture angles do not overlap. In addition, a plurality of communication devices 70 are arranged in the real space RS.

[0185] Returning to FIG. 10, the description will continue.

[0186] In this embodiment, a person P who is a management target of the information management system 1 is wearing a mobile terminal 30B instead of a mobile terminal 30. In this embodiment, the person P wearing the mobile terminal 30B will be referred to as the second person P2. As in the above embodiment, the second person P2 is the person wearing the mobile terminal 30B. The second person P2 is an example of a person P. As in the above embodiment, the person P who appears in a captured image captured by the imaging device 20 will be referred to as the first person P1. Therefore, when a person P who exists in the real space RS appears in a captured image, the person P who appears in the captured image will be referred to as the first person P1, and when the person P is wearing the mobile terminal 30B, the person P will be referred to as the second person P2. Therefore, the first person P1 and the second person P2 may be the same person P.

[0187] The mobile terminal 30B includes a communication unit 31, a UI unit 32, a storage unit 33, a sensor 34, a drive unit 35, and a processing unit 37. The communication unit 31, the UI unit 32, the storage unit 33, the sensor 34, the drive unit 35, and the processing unit 37 are connected to each other so as to be able to communicate with each other via a bus or the like.

[0188] The mobile terminal 30B is similar to the mobile terminal 30 of the above embodiment, except that it includes a processing unit 37 instead of the processing unit 36. The communication unit 31 of the mobile terminal 30B functions as a communication interface for communicating with the information processing system 10 via the network NW, and also has the function of receiving and transmitting wireless signals transmitted from the communication device 70. The communication unit 31 may transmit and receive wireless signals based on a wireless communication standard such as a short-range wireless communication standard. The wireless communication standard used by the communication unit 31 is not limited to Bluetooth. For example, the communication unit 31 may perform wireless communication using an infrared communication standard such as IrDA (Infrared Data Association).

[0189] The processing unit 37 executes information processing in the mobile terminal 30B. The mobile terminal 30B is realized by one or more processors. For example, the processing unit 37 may be realized by causing a processor such as a CPU to execute a program, i.e., by software. Furthermore, each of the one or more processing function units included in the processing unit 37 may be realized by a processor such as a dedicated IC, i.e., by hardware. Each of the units may be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or may realize two or more of the units.

[0190] The processing unit 37, like the processing unit 36 ​​in the above embodiment, sequentially transmits the sensing result information of the second person P2 obtained by the sensor 34 to the information processing system 10B.

[0191] As described in the above embodiment, the sensing result information includes at least the sensing result of the sensor 34 and the terminal ID of the mobile terminal 30B equipped with the sensor 34. The sensing result information includes at least one of the following: angular velocity, acceleration, and temperature detected by an IMU sensor, which is an example of the first sensor 34A; biological information such as heart rate, body temperature, blood oxygen concentration, blood pressure, and sleep cycle detected by a sensor that detects biological information, which is an example of the first sensor 34A; and information such as pressure distribution on the soles of the feet of the second person P2, temperature distribution on the soles of the feet, and sweat distribution on the soles of the feet, which are detected by the second sensor 34B.

[0192] In this embodiment, the processing unit 37 calculates the location information of the mobile terminal 30B in which the processing unit 37 is installed based on the location information of one or more communication devices 70 that emit wireless signals and the wireless signals received from the communication devices 70.

[0193] The processing unit 37 acquires map data 18E from the information processing system 10B. The map data 18E is map data of the real space RS. The map data 18E registers the position information and communication device ID of each of the multiple communication devices 70 arranged in the real space RS, and the image capturing device ID and position information of each of the multiple image capturing devices 20. The map data 18E also registers the image capturing direction and image capturing angle of view of each of the multiple image capturing devices 20. The map data 18E is map data generated, for example, by BIM (Building Information Modeling), 2D CAD (Two-dimensional Computer Aided Design), 3D CAD (Three-dimensional Computer Aided Design), or the like.

[0194] The processing unit 37 acquires the map data 18E by receiving the map data 18E from the information processing system 10B. Alternatively, the processing unit 37 may acquire the map data 18E by storing the map data 18E in advance in the storage unit 33 and reading the map data 18E from the storage unit 33. Alternatively, the processing unit 37 may acquire the map data 18E from the information processing system 10B or the storage unit 33, the map data 18E including at least the area in which the mobile terminal 30B is located.

[0195] Furthermore, the processing unit 37 receives wireless signals transmitted from one or more communication devices 70 arranged in the real space RS. Then, based on the received wireless signals, the processing unit 37 calculates the position information of the mobile terminal 30B equipped with the processing unit 37 by a known method.

[0196] Specifically, assume that the processing unit 37 receives wireless signals from three or more communication devices 70. In this case, the processing unit 37 calculates the distance between the mobile terminal 30B and the communication devices 70 using a known method based on at least one of the reception strength and the angle of arrival (AoA) of the received wireless signals. The processing unit 37 also acquires the location information of the communication devices 70 by reading, from the map data 18E, the location information of the communication devices 70 identified by the communication device IDs included in the received wireless signals. The processing unit 37 then calculates the location information of the mobile terminal 30B by inverse calculation using the principle of triangulation using the location information of each communication device 70 that received the wireless signals and the distance between each communication device 70 and the mobile terminal 30B.

[0197] Furthermore, when the processing unit 37 receives wireless signals from one to two communication devices 70 and the location information can be calculated by correction, the processing unit 37 calculates the location information of the mobile terminal 30B using the following method. Specifically, the processing unit 37 calculates the distance between the mobile terminal 30B and the communication devices 70 using a known method based on the reception strength of the received wireless signals, as described above. The processing unit 37 also acquires the location information of the communication devices 70 by reading, from the map data 18E, the location information of the communication devices 70 identified by the communication device IDs included in the received wireless signals. Then, when the processing unit 37 can calculate the location information of the mobile terminal 30B using a known method using the location information of each communication device 70 that received the wireless signals, the distance between each communication device 70 and the mobile terminal 30B, and at least one of the constraints and the complementary information, the processing unit 37 calculates the location information of the mobile terminal 30B using this calculation.

[0198] The constraint conditions are, for example, information that indicates the range in which the mobile terminal 30B can move, areas such as walls where the mobile terminal 30B cannot move, and the upper limit of the movement speed of the person P carrying the mobile terminal 30B, which are shown in the map data 18E. The constraint conditions may be registered in the map data 18E. The supplementary information is the sensing results of angular velocity, acceleration, and the like detected by an IMU sensor, which is an example of the first sensor 34A.

[0199] Furthermore, when processing unit 37 receives wireless signals from one to two communication devices 70 and determines that the location information cannot be calculated by correction, processing unit 37 treats the location information of mobile terminal 30B as not being able to be calculated at the current calculation timing. In this case, processing unit 37 may use the previously calculated location information of mobile terminal 30B as the currently calculated location information of mobile terminal 30B.

[0200] Each time the processing unit 37 calculates the position information of the mobile terminal 30B in which the processing unit 37 is installed, the processing unit 37 transmits the terminal ID of the mobile terminal 30B and the position information of the mobile terminal 30B to the information processing system 10B.

[0201] In addition, each time the processing unit 37 calculates the location information of the mobile terminal 30B in which the processing unit 37 is installed, the processing unit 37 may display on the UI unit 32 a display image in which an image representing the calculated location information of the mobile terminal 30B is superimposed on a map image represented by the map data 18E.

[0202] Next, the information processing system 10B will be described.

[0203] The information processing system 10B is one or more information processing devices, and is configured by one or more dedicated or general-purpose computers.

[0204] The information processing system 10B includes a communication unit 11, an input unit 12, a display unit 13, a storage unit 18, and a processing unit 17. The communication unit 11, the input unit 12, the storage unit 18, the storage unit 14, and the processing unit 17 are communicatively connected via a bus or the like.

[0205] The communication unit 11, the input unit 12, and the display unit 13 are the same as those in the information processing system 10 of the above embodiment. In this embodiment, the communication unit 11 communicates with the mobile terminal 30B instead of the mobile terminal 30. The communication unit 11 may also be capable of communicating with the communication device 70 via the network NW.

[0206] The processing unit 15 stores various types of data. The storage unit 18 is, for example, a semiconductor memory element such as a RAM or a flash memory, a hard disk, an optical disk, or the like. The storage unit 18 may be a storage device provided outside the information processing system 10B. Furthermore, at least a portion of the information stored in the storage unit 18 may be stored in a storage device provided outside the information processing system 10B. Furthermore, the storage unit 18 may be a storage medium that stores or temporarily stores programs and various types of information that have been downloaded via a LAN, the Internet, or the like.

[0207] In this embodiment, the storage unit 18 stores information such as photography management information 18A, wearing management information 18B, personal management information 18C, correspondence management information 14D, and map data 18E. The correspondence management information 14D is the same as in the above embodiment. Details of the photography management information 18A, wearing management information 18B, and personal management information 18C will be described later.

[0208] At least one of the display unit 13, the storage unit 18, and an external device such as a server device connected to the information processing system 10B via the network NW functions as an output device 16 that outputs multimodal information, which will be described later.

[0209] The processing unit 17 executes information processing in the information processing system 10B.

[0210] The processing unit 17 includes an acquisition unit 15A, an imaging device identification unit 17H, an identification unit 17B, a determination unit 17I, a derivation unit 15C, a comparison unit 15D, a correspondence unit 17E, a derivation unit 17J, and an output control unit 15F.

[0211] The acquisition unit 17A, the imaging device identification unit 17H, the identification unit 17B, the determination unit 17I, the derivation unit 15C, the comparison unit 15D, the association unit 17E, the derivation unit 17J, and the output control unit 15F are realized by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU execute a program, i.e., by software. Each of the above units may be realized by a processor such as a dedicated IC, i.e., by hardware. Each of the above units may be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or two or more of the units.

[0212] Furthermore, at least one of the above-mentioned units included in the processing unit 17 may be mounted on an external information processing device such as an external system communicatively connected to the information processing system 10B via a network NW or the like.

[0213] The comparison unit 15D and the output control unit 15F are the same as those in the above embodiment. The processing unit 17 may further include the operation control unit 15G described in the above embodiment.

[0214] The acquisition unit 17A acquires captured images and sensing result information in the same manner as the acquisition unit 15A in the above embodiment. As in the above embodiment, the acquisition unit 17A acquires captured images captured by the imaging device 20 and sensing result information by the sensor 34 of the mobile terminal 30B. The acquisition unit 17A sequentially accepts the captured images from the imaging device 20 and sequentially accepts the sensing result information from the mobile terminal 30B. In this embodiment, the acquisition unit 17A further sequentially accepts the location information and terminal ID of the mobile terminal 30B calculated by the mobile terminal 30B from the mobile terminal 30B.

[0215] The acquisition unit 17A acquires captured images, sensing result information, and location information of the mobile terminal 30B with timestamps having the same shooting timing, sensing timing, and calculation timing as captured images, sensing result information, and location information of the mobile terminal 30B with the same acquisition time. In this embodiment, the sensing timing, shooting timing, and calculation timing of the location information of the mobile terminal 30B may be referred to as acquisition time.

[0216] The acquisition unit 17A stores the acquired photographed image in the photography management information 18A in association with the acquisition time, which is the timing at which the photographed image was taken.

[0217] Furthermore, the acquisition unit 17A stores the acquired sensing result information in the wearing management information 18B in association with the acquisition time, which is the sensing timing of the sensing result information, and the terminal ID included in the sensing result information. The acquisition unit 17A also stores the acquired position information of the mobile terminal 30B in association with the acquisition time, which is the calculation timing of the position information, and the terminal ID information acquired together with the position information, in the wearing management information 18B. Therefore, a second movement trajectory represented by the transition of the position information of the mobile terminal 30B is registered for each mobile terminal 30B in the wearing management information 18B. The second movement trajectory is information representing the movement trajectory represented by the transition of the position information of the mobile terminal 30B calculated by the mobile terminal 30B.

[0218] 12 is a schematic diagram showing an example of the data structure of the photography management information 18A. Similar to the photography management information 14A in the above embodiment, the photography management information 18A is a database for managing information related to photographed images photographed by the photography device 20. The data format of the photography management information 18A is not limited to a database.

[0219] The photography management information 14A is a database that associates a photographed image ID, a photography device ID, an acquisition time, a photographed image, a first person ID, first analysis result information, a first estimated motion, and at least one of emotion information and interest level information. That is, the photography management information 18A is a database that further associates at least one of emotion information and interest level information with the photography management information 14A described in the above embodiment.

[0220] In this embodiment, the first analysis result information includes first attribute information representing the attributes of the first person P1. As described in the above embodiment, the first analysis result information is information representing the analysis results obtained by analyzing a corresponding captured image of the first person P1 who appears in the captured image. The first attribute information is an example of attribute information representing the attributes of the person P, and is attribute information obtained by analyzing the captured image. The attribute information includes information relating to multiple types of attributes, such as gender, age, height, and weight.

[0221] As described in the above embodiment, the first estimated motion is an estimated motion of the first person P1 identified by the corresponding first person ID, estimated by analyzing the corresponding captured image. In this embodiment, the first estimated motion includes position information of the first person P1 in addition to the estimated motion of the first person P1. That is, the first estimated motion includes a first movement trajectory representing the movement trajectory of the first person P1 included in the captured image, obtained by analyzing one or more captured images using a known analysis method. The first movement trajectory is a trajectory of the positional movement of the first person P1, obtained by analyzing the captured image using a known analysis method. Therefore, the first movement trajectory is represented by a group of position information associated with different acquisition times of the first person P1 identified by the same first person ID in the photography management information 18A.

[0222] The acquisition unit 17A stores photographed images acquired from the photographing device 20 in photographing management information 18A in association with the photographing ID of the photographing device 20 that captured the photographed image and the acquisition time, which is the timing at which the photographed image was captured. Furthermore, the acquisition unit 17A assigns a photographed image ID to the photographed image for each photographing device ID and acquisition time, and registers the photographed image in the photographing management information 18A. Therefore, each time a new photographed image is captured by the photographing device 20, the photographed image and the photographed image ID are registered in the photographing management information 18A in association with the photographing device ID of the photographing device 20 that captured the photographed image and the acquisition time, which is the timing at which the photographed image was captured. Furthermore, photographed images captured by the photographing device 20 included in the information management system 1B are sequentially registered in the photographing management information 18A in association with each of a plurality of different acquisition times in chronological order.

[0223] The first person ID, the first analysis result information, and the first estimated action are derived by the processing unit 17 through the same process as in the above embodiment, and are registered in the photography management information 18A (details will be described later). At least one of the emotion information and the interest level information is also homogenized through the process to be described later by the processing unit 17, and is registered in the photography management information 18A (details will be described later).

[0224] 13 is a schematic diagram of an example of the data configuration of the wearing management information 18B. Similar to the wearing management information 14B, the wearing management information 18B is a database for managing information about the portable terminal 30B worn by the second person P2. In other words, the wearing management information 18B is a database for managing information about the second person P2 wearing the portable terminal 30B. The data format of the wearing management information 18B is not limited to a database.

[0225] The wearing management information 18B is a database that associates a terminal ID, a second person ID, an acquisition time, sensing result information, second analysis result information, a first estimated motion, and at least one of emotion information and interest level information. That is, the wearing management information 18B is a database that further associates at least one of emotion information and interest level information in addition to the wearing management information 14B described in the above embodiment.

[0226] The terminal ID and the second person ID are the same as those in the above embodiment. In the wearing management information 18B, for each of the second persons P2 who are managed by the information management system 1B, the second person ID of the second person P2 and the terminal ID of the portable terminal 30B that is worn by or will be worn by the second person P2 are registered in advance in association with each other.

[0227] The acquisition time, sensing result information, and second analysis result information registered in the wearing management information 18B are the same as the acquisition time in the wearing management information 14B in the above embodiment. As described in the above embodiment, the second estimated motion is an estimated motion of the second person P2 identified by the corresponding second person ID, estimated by analyzing the corresponding sensing result information. In this embodiment, the second estimated motion includes the position information of the portable terminal 30B in addition to the estimated motion of the second person P2. That is, the second estimated motion further includes the position information of the portable terminal 30B calculated by the portable terminal 30B as the position information of the second person P2. That is, the second estimated motion includes a second movement trajectory representing the movement trajectories of the portable terminal 30B and the second person P2 wearing the portable terminal 30B, represented by a group of position information of the portable terminal 30B calculated by the portable terminal 30B. The second movement trajectory is a trajectory of the position movement of the second person P2 wearing the portable terminal 30B, represented by a group of position information of the portable terminal 30B.

[0228] The acquisition unit 17A stores the sensing result information acquired from the portable terminal 30B in the wearing management information 18B in association with the terminal ID of the portable terminal 30B equipped with the sensor 34 that detected the sensing result information and the acquisition time, which is the sensing timing, of the sensing result information. Therefore, each time new sensing result information is acquired by the sensor 34 equipped in the portable terminal 30B, the sensing result information is registered in association with the terminal ID of the portable terminal 30B equipped with the sensor 34 that detected the sensing result information, the acquisition time, which is the sensing timing, and the second person ID of the second person P2 wearing the portable terminal 30B. Furthermore, the sensing result information detected by the sensor 34 equipped in the portable terminal 30B included in the information management system 1B is sequentially registered in the wearing management information 18B in association with each of a plurality of acquisition times that are different from one another in chronological order. In addition, the position information of the mobile terminal 30B calculated by the mobile terminal 30B is sequentially registered in the wearing management information 18B in correspondence with the terminal ID of the mobile terminal 30B and the acquisition time, which is the calculation time of the position information.

[0229] The second analysis result information and the second estimated motion are derived by processing by the processing unit 17, as in the above embodiment, and are registered in the wearing management information 18B (described in detail later).

[0230] As in the above embodiment, the storage unit 18 further stores personal management information 18C.

[0231] 14 is a schematic diagram of an example of the data configuration of the personal management information 18C. Similar to the personal management information 14C, the personal management information 18C is a database for managing detailed information about the second person P2 who is a target of management by the information management system 1B. The data format of the personal management information 18C is not limited to a database.

[0232] The personal management information 18C is a database that associates the second person ID of the second person P2 with the person detailed information of the second person P2 identified by the second person ID. The second person ID is the same as the second person ID registered in the wearing management information 18B. That is, like the wearing management information 18B, the personal management information 18C pre-registers the second person ID of each of the multiple second persons P2 that are managed by the information management system 1B.

[0233] The person detailed information is information about the second person P2 and preferably includes information that cannot be estimated from the captured image and the sensing result information. For example, the person detailed information includes the name, second attribute information, physical information, etc. of the second person P2. The second attribute information is information about the attributes of the second person P2. The second attribute information is an example of attribute information that represents the attributes of the person P, and is attribute information obtained by analyzing the sensing result information obtained from the mobile terminal 30B, the location information of the mobile terminal 30B, etc. As described above, the attribute information includes information about multiple types of attributes such as gender, age, height, and weight. The physical information is information that represents the physical characteristics of the second person P2. Note that the person detailed information may be any information about the second person P2, and may further include information other than the name, second attribute information, and physical information.

[0234] Returning to FIG. 10, the description will continue.

[0235] By adjusting the timestamps, the acquisition unit 17A acquires the captured images, sensing result information, and location information of the portable terminal 30B at the same acquisition times (capturing timing, sensing timing, calculation timing) from each of the photographing device 20 and the portable terminal 30B. Furthermore, the acquisition unit 17A may store the captured images, sensing result information, and location information of the portable terminal 30B acquired from each of the photographing device 20 and the portable terminal 30B in the photographing management information 18A and the wearing management information 18B, respectively, and then sequentially acquire the captured images, sensing result information, and location information of the portable terminal 30B at the same acquisition times (capturing timing, sensing timing, calculation timing) from the photographing management information 18A and the wearing management information 18B in chronological order. In addition, when acquiring the captured image, sensing result information, and location information of the mobile terminal 30B from the photography management information 18A and the wearing management information 18B, the acquisition unit 17A only needs to sequentially acquire, from the captured image, sensing result information, and location information of the mobile terminal 30B that have not yet been processed, the captured image, sensing result information, and location information of the mobile terminal 30B that have the same acquisition time and that correspond to the acquisition time next (further in the future, more recent) to the acquisition time corresponding to the previously acquired captured image, sensing result information, and location information of the mobile terminal 30B.

[0236] The photographing device identification unit 17H identifies the photographing device 20 that captures the photographed image in which the wearer of the portable terminal 30B appears, based on the position information of the portable terminal 30B acquired by the acquisition unit 17A. Specifically, the acquisition unit 17A uses a known method to identify the photographing range of each photographing device 20 in the real space RS represented by the map data 18E, based on the map data 18E and the position information, photographing direction, and photographing angle of view of each photographing device 20 registered in the map data 18E. The photographing device identification unit 17H then identifies the photographing device 20 within the photographing range that includes the position information of the portable terminal 30B acquired by the acquisition unit 17A. Through these processes, the photographing device identification unit 17H identifies the photographing device 20 that captures the photographed image in which the wearer of the portable terminal 30B appears. In other words, the photographing device identification unit 17H identifies the photographing device 20 that captures the photographed image in which the second person P2, the wearer of the portable terminal 30B, appears as the first person P1.

[0237] Similar to the identification unit 15B in the above embodiment, the identification unit 17B identifies a first person P1 appearing in a photographed image acquired by the acquisition unit 17A. However, in the present embodiment, the identification unit 17B performs image analysis of the photographed image acquired by the acquisition unit 17A and captured by the photographing device 20 identified by the photographing device identification unit 17H using a known method, thereby identifying each of one or more persons P appearing in the photographed image as a first person P1. Similar to the identification unit 15B, the identification unit 17B assigns a first person ID to the identified first person P1 and registers the assigned first person ID in the photographing management information 18A in association with the photographed image and the photographed image ID at the acquisition time.

[0238] The determination unit 17I determines whether or not a plurality of first persons P1 appear in the captured image acquired by the acquisition unit 17A and captured by the image capturing device 20 identified by the image capturing device identification unit 17H.

[0239] If the determination unit 17I determines that only one first person P1 appears in the captured image, the association unit 17E associates the first information about the first person P1 with the second information obtained from the mobile terminal 30B.

[0240] The definitions of the first information and the second information are the same as those in the above embodiment.

[0241] Specifically, the first information is information related to the first person P1. Specifically, the first information includes first analysis result information of the first person P1 appearing in the captured image. The first information may further include a first estimated motion derived from the first analysis result information. The first information may further include a captured image appearing in the first person P1. In this embodiment, the first information may further include at least one of emotion information and interest level information of the first person P1. The emotion information and interest level information are derived by a derivation unit 17J (described in detail below).

[0242] For this reason, as shown in FIG. 12 , in this embodiment, the first information includes a captured image identified by a captured image ID, the photographing device 20 that captured the captured image identified by a photographing device ID, the acquisition time, first analysis result information including first attribute information, a first estimated action including a first movement trajectory, and at least one of emotion information and interest level information.

[0243] The second information includes information acquired from the portable device 30B worn by the second person 2P who is the wearer. In other words, the second information is information about the second person P2 who is the wearer of the portable device 30B. In particular, the second information includes second analysis result information about the second person P2 who is the wearer of the portable device 30B equipped with the sensor 34 that detected the sensing result information. The second analysis result information includes data received from the sensor 34. The second analysis result information may also include an analysis result of the sensing result by the sensor 34. The second information may also include a second estimated motion derived from the second analysis result information. The second analysis result information includes a second movement trajectory. The second information may also include sensing result information of the second person P2. The second information may also include person detail information of the second person P2. In this embodiment, the second information may also include at least one of emotion information and interest level information of the second person P2. The emotion information and interest level information are derived by a derivation unit 17J (described in detail later).

[0244] Therefore, as shown in Figures 13 and 14, in this embodiment, the second information includes the mobile terminal 30B identified by the terminal ID, the acquisition time, the sensor result information, the second analysis result information, the second estimated motion including the second movement trajectory, at least one of the emotion information and the interest level information, the name of the second person P2, the second attribute information of the second person P2, and the physical information of the second person P2.

[0245] Returning to FIG. 10, the description will continue.

[0246] Similar to the correspondence unit 15E, the correspondence unit 17E corresponds the first information about the first person P1 with the second information about the second person P2 by correlating the first person ID of the first person P1 with the second person ID of the second person P2 and registering them in the correspondence management information 14D (see Figure 7).

[0247] On the other hand, if the judgment unit 17I determines that multiple first persons P1 appear in the captured image acquired by the acquisition unit 17A and captured by the photographing device 20 identified by the photographing device identification unit 17H, it performs the following processing.

[0248] First, the determination unit 17I derives first attribute information of each of the first persons P1 included in the captured image by analyzing the area in which each of the first persons P1 appears in the captured image using a known method. Furthermore, the determination unit 17I reads second attribute information associated with the terminal ID of the portable terminal 30B that transmitted the sensing result information acquired by the acquisition unit 17A at the same acquisition time as the captured image from the personal management information 18C (see FIG. 14 ), thereby deriving second attribute information of each of the second persons P2 who wear the portable terminal 30B.

[0249] Then, the determination unit 17I determines whether there are first persons P1 and second persons P2 whose first attribute information and second attribute information match. As described above, the attribute information includes information on multiple types of attributes, such as gender, age, height, and weight. The determination unit 17I determines whether there are first persons P1 and second persons P2 whose first attribute information and second attribute information match by determining whether at least one or more types of attributes included in the derived first attribute information and second attribute information match.

[0250] If the first attribute information of the first person P1 derived based on the captured image matches the second attribute information of the second person P2 who is the wearer of the mobile terminal 30B and which is stored in advance in correspondence with the second person ID of the second person P2, the matching unit 17E matches the first information of the first person P1 whose first attribute information matches with the second information of the second person P2 whose second attribute information matches with the attribute information.

[0251] The correspondence unit 17E corresponds the first information about the first person P1 with the second information about the second person P2 by correlating the first person ID of the first person P1 and the second person ID of the second person P2, whose attribute information matches, and registering them in the correspondence management information 14D (see Figure 7).

[0252] On the other hand, it is assumed that the determination unit 17I determines that the first attribute information and the second attribute information do not match.

[0253] In this case, the derivation unit 15C derives a first estimated motion of the first person P1 appearing in the captured image in the same manner as in the above embodiment, except that the derivation unit 15C uses the captured image acquired by the acquisition unit 17A and captured by the camera device 20 identified by the camera device identification unit 17H. Also, the derivation unit 15C derives a second estimated motion of the second person P2 based on sensing result information by the sensor 34 acquired by the acquisition unit 15A at the same acquisition time as the captured image, in the same manner as in the above embodiment.

[0254] The derivation unit 15C then registers the first estimated motion estimated for each first person P1, together with the first analysis result information used to derive the first estimated motion, in the photography management information 18A in association with the first person ID of the first person P1 and the captured image ID of the captured image used to derive the first analysis result information (see FIG. 12 ). The derivation unit 15C also registers the second estimated motion derived for each second person P2 in association with the sensing result information and second analysis result information used to derive the second estimated motion in the wearing management information 18B (see FIG. 13 ).

[0255] The comparison unit 15D is similar to that in the above embodiment. That is, the comparison unit 15D compares, for each captured image at the same acquisition time, a first estimated motion of a first person P1 appearing in the captured image with a second estimated motion of a second person P2 derived based on the sensing result information. Then, the comparison unit 15D determines whether the first estimated motion and the second estimated motion match.

[0256] The comparison unit 15D may compare, for each capture time, a first estimated motion of a first person P1 and a first movement trajectory of the first person P1 that appear in the captured image with a second estimated motion of a second person P2 and a second movement trajectory of the second person P2 that are derived based on the sensing result information. Then, the comparison unit 15D may determine whether the first estimated motion and the second estimated motion match and whether the first movement trajectory and the second movement trajectory match.

[0257] Assume a scene in which multiple first persons P1 appear in a captured image acquired at a certain capture time, and a first inferred motion is derived for each of the multiple first persons P1. In this case, the comparison unit 15D compares the first inferred motion of each of the multiple first persons P1 with the second inferred motion of each of the multiple second persons P2 that is derived from the sensing result information of each of the multiple second persons P2 that are managed by the information management system 1.

[0258] If the first estimated action and the second estimated action match based on the comparison result between the first estimated action and the second estimated action by the comparison unit 15D, the matching unit 15E matches the first information about the first person P1 appearing in the captured image with the second information obtained from the mobile terminal 30.

[0259] The matching unit 15E matches the first information about the first person P1 and the second information about the second person P2 whose estimated actions match by matching the first person ID of the first person P1 and the second person ID of the second person P2 whose estimated actions match and registering them in the matching management information 14D (see Figure 7).

[0260] As a result of the registration process to the correspondence management information 14D by the correspondence unit 15E, the first information corresponding to the first person ID registered in correspondence with the correspondence management information 14D and the second information corresponding to the second person ID associated with the first person ID in the correspondence management information 14D are associated with each other.

[0261] The derivation unit 17J derives at least one of the emotional information and interest level information of the first person P1 appearing in the captured image by analyzing the captured image acquired by the acquisition unit 17A and captured by the imaging device 20 identified by the imaging device identification unit 17H using a known method.

[0262] The emotion information is information that represents the emotion of the first person P1, specifically, joy, excitement, sadness, anger, surprise, neutral, etc.

[0263] For example, the derivation unit 17J analyzes the facial area of ​​the first person P1 included in the captured image, analyzes the facial expression expressed by the face in the facial area, and derives emotional information representing the emotion that is previously associated with the analyzed facial expression.

[0264] The interest level information is information that indicates the subject of interest of person P and the degree of interest in that subject.

[0265] For example, the derivation unit 17J identifies the viewing direction of the first person P1 based on skeletal information of the first person P1 appearing in the captured image, and derives an object present in the viewing direction as an object of interest using at least one of the captured image and the map data 18E. The derivation unit 17J also calculates the time the first person P1 continuously views the object by analyzing the continuously acquired captured images in chronological order. The longer the time, the higher the degree of interest in the object the derivation unit 17J derives. The derivation unit 17J may also derive a higher degree of interest in the object based on the skeletal information, the greater the inclination of at least a part of the first person P1's body toward the object or the closer the distance between the first person P1 and the object. Furthermore, the derivation unit 17J may derive the degree of interest by determining, based on the skeletal information, which of predetermined interest levels the gesture of the first person P1 corresponds to.

[0266] Then, the derivation unit 17J registers at least one of the derived emotion information and interest level information of the first person P1 in the photography management information 18A in association with the first person ID of the first person P1 (see FIG. 12 ). Therefore, as shown in FIG. 12 , at least one of the emotion information and interest level information of the first person P1 derived from the photographed image is registered as first information in the photography management information 18A.

[0267] Returning to FIG. 10, the description will continue.

[0268] In addition, the derivation unit 17J derives at least one of the emotional information and interest level information of the second person P2 who is wearing the mobile terminal 30B using a known method based on the angular velocity and acceleration, which are the detection results of the IMU sensor, and biological information such as heart rate, body temperature, blood oxygen concentration, and blood pressure, which are included in the sensing result information acquired by the acquisition unit 17A from the mobile terminal 30B.

[0269] Note that the mobile terminal 30B may derive at least one of the emotion information and the interest level information of the second person P2 based on the sensing result information. In this case, the derivation unit 17J may derive at least one of the emotion information and the interest level information of the second person P2 by receiving at least one of the emotion information and the interest level information from the mobile terminal 30B.

[0270] Then, the derivation unit 17J registers at least one of the derived emotion information and interest level information of the second person P2 in the wearing management information 18B in association with the second person ID of the second person P2 (see FIG. 13 ). Therefore, as shown in FIG. 13 , at least one of the emotion information and interest level information of the second person P2 derived based on information obtained from the mobile terminal 30B is registered as second information in the wearing management information 18B.

[0271] Therefore, in the information processing system 10B of this embodiment, it is possible to integrate and manage multimodal information that associates, for each person P, multiple types of information related to that person P, which further includes at least one of emotional information and interest level information.

[0272] As in the above embodiment, the output control unit 15F outputs multimodal information in which the first information and the second information are associated with each other to the output device 16. By outputting the multimodal information to the output device 16, the output control unit 15F can output multimodal information for each person P.

[0273] Next, an example of the flow of information processing executed by the information management system 1B of this embodiment will be described.

[0274] First, an example of the flow of the location information calculation process executed by the processing unit 37 of the mobile terminal 30B will be described.

[0275] FIG. 15 is a flowchart showing an example of the flow of the process of calculating the position information of the mobile terminal 30B by the processing unit 37 of the mobile terminal 30B.

[0276] The processing unit 37 of the portable terminal 30B sequentially transmits the sensing result information of the second person P2 obtained by the sensor 34 to the information processing system 10B. Furthermore, the processing unit 37 calculates the position information of the portable terminal 30B in which the processing unit 37 is installed by the following process, and transmits the position information to the information processing system 10B.

[0277] Specifically, the processing unit 37 acquires the map data 18E from the information processing system 10B (step S100).

[0278] Next, the processing unit 37 receives wireless signals transmitted from one or more communication devices 70 arranged in the real space RS (step S102).

[0279] The processing unit 37 determines whether or not wireless signals have been received from three or more communication devices 70 in step S102 (step S104). The processing unit 37 makes the determination in step S104 by determining whether or not three or more types of wireless signals containing different communication device IDs have been received.

[0280] If the determination in step S104 is affirmative (step S104: Yes), the process proceeds to step S106. In step S106, the processing unit 37 calculates the location information of the mobile terminal 30B based on the reception strength of the received wireless signal and the location information of the communication device 70 registered in the map data 18E acquired in step S100 (step S106). For example, the processing unit 37 calculates the distance between the mobile terminal 30B and the communication device 70 using a known method based on the reception strength of the received wireless signal. The processing unit 37 also acquires the location information of the communication device 70 by reading, from the map data 18E, the location information of the communication device 70 identified by the communication device ID included in the received wireless signal. The processing unit 37 then calculates the location information of the mobile terminal 30B by inverse calculation applying the principle of triangulation using the location information of each communication device 70 that received the wireless signal and the distance between each communication device 70 and the mobile terminal 30B.

[0281] The processing unit 37 then causes the UI unit 32 to display an image representing the calculated position information of the captured image 40, superimposed on the map of the map data 18E acquired in step S100 (step S108).Then, the process proceeds to step S118, which will be described later.

[0282] On the other hand, if the determination in step S104 is negative (step S104: No), the processing proceeds to step S110. In step S110, the processing unit 37 determines whether the processing unit 37 has received wireless signals from one to two communication devices 70 and whether the position information can be calculated by correction (step S110).

[0283] If the determination in step S110 is affirmative (step S110: Yes), the processing proceeds to step S112. In step S112, the processing unit 37 calculates the location information of the mobile terminal 30B based on the reception strength of the received wireless signal, the location information of the communication device 70, and correction processing (step S112). In step S112, the processing unit 37 calculates the distance between the mobile terminal 30B and the communication device 70 using a known method based on the reception strength of the received wireless signal. The processing unit 37 also acquires the location information of the communication device 70 by reading, from the map data 18E, the location information of the communication device 70 identified by the communication device ID included in the received wireless signal. The processing unit 37 then calculates the location information of the mobile terminal 30B using a known method based on the location information of each communication device 70 that received the wireless signal, the distance between each communication device 70 and the mobile terminal 30B, and at least one of the constraint condition and the complementary information.

[0284] The processing unit 37 then superimposes an image representing the calculated position information of the captured image 40 on the map of the map data 18E acquired in step S100 and displays it on the UI unit 32 (step S114).Then, the process proceeds to step S118, which will be described later.

[0285] On the other hand, if the determination in step S110 is negative, the process proceeds to step S116. In step S116, the processing unit 37 assumes that the location information of the mobile terminal 30B could not be calculated at the current calculation timing (step S116). In this case, the processing unit 37 may use the previously calculated location information of the mobile terminal 30B as the currently calculated location information of the mobile terminal 30B (step S116). Then, the process proceeds to step S118.

[0286] In step S118, it is determined whether the location information of the mobile terminal 30B has been calculated (step S118). If the determination in step S118 is negative (step S118: No), the process returns to step S102. If the determination in step S118 is positive (step S118: Yes), the process proceeds to step S120.

[0287] In step S120, the processing unit 37 transmits the terminal ID of the mobile terminal 30B and the calculated location information of the mobile terminal 30B to the information processing system 10B (step S120).

[0288] Next, the processing unit 37 determines whether to end the process (step S122). For example, the processing unit 37 executes the determination of step S122 by determining whether an instruction to end the transmission of the location information has been received from the UI unit 32. If the determination in step S122 is negative (step S122: No), the processing unit 37 returns to step S102. If the determination in step S122 is positive (step S122: Yes), the processing unit 37 ends this routine.

[0289] Next, an example of the flow of information processing executed by the information processing system 10B of this embodiment will be described.

[0290] FIG. 16 is a flowchart showing an example of the flow of information processing executed by the information processing system 10B of this embodiment.

[0291] The acquiring unit 17A of the information processing system 10B acquires the captured image, the sensing result information, and the location information of the mobile terminal 30B (step S200). The acquiring unit 17A acquires, from the photographing device 20 and the mobile terminal 30B, the captured image captured at the same acquisition time by the photographing device 20, the sensing result information by the sensor 34 of the mobile terminal 30B, and the location information and the terminal ID of the mobile terminal 30B calculated by the mobile terminal 30B.

[0292] The acquisition unit 17A stores the captured image, sensing result information, and location information of the mobile terminal 30B acquired in step S200 in the photography management information 18A and the wearing management information 18B, respectively (step S202). Specifically, the acquisition unit 17A stores the acquired captured image in the photography management information 18A in association with the acquisition time, which is the timing at which the captured image was captured. The acquisition unit 17A also stores the acquired sensing result information in the wearing management information 18B in association with the acquisition time, which is the sensing timing of the sensing result information, and the terminal ID included in the sensing result information. The acquisition unit 17A also stores the acquired location information of the mobile terminal 30B in association with the acquisition time, which is the timing at which the location information was calculated, and the terminal ID information acquired together with the location information, in the wearing management information 18B.

[0293] The photographing device specifying unit 17H specifies the photographing device 20 that captures the photographed image in which the wearer of the portable terminal 30B appears, based on the position information of the portable terminal 30B acquired in step S200 (step S204).

[0294] The identification unit 17B performs image analysis of the photographed images acquired in step S200 by the photographing device 20 identified in step S204 using a known method, thereby identifying each of one or more persons P appearing in the photographed images as a first person P1 (step S206). The identification unit 17B assigns a first person ID to the first person P1 identified in step S206, and registers the assigned first person ID in the photographing management information 18A in association with the photographed image and the photographed image ID at the acquisition time (step S208).

[0295] Next, the determination unit 17I determines whether or not a plurality of first persons P1 appear in the captured image acquired in step S200 and captured by the image capturing device 20 identified in step S204 (step S210).

[0296] If the determination unit 17I determines that only one first person P1 appears in the captured image (step S210: No), the association unit 17E associates the first information about the first person P1 with the second information acquired from the portable terminal 30B (step S212). The association unit 17E associates the first information about the first person P1 with the second information about the second person P2 by registering the first person ID of the first person P1 and the second person ID associated with the terminal ID of the portable terminal 30B in the association management information 14D (see FIG. 7). Then, the process proceeds to step S226, which will be described later.

[0297] On the other hand, if the judgment unit 17I determines in step S210 that multiple first persons P1 appear in the captured image acquired in step S200 and captured by the photographing device 20 identified in step S204 (step S210: Yes), the process proceeds to step S214.

[0298] In step S214, the determination unit 17I determines whether there are first persons P1 and second persons P2 whose first attribute information and second attribute information match (step S214). The determination unit 17I derives first attribute information for each first person P1 included in the captured image by analyzing the area in which each first person P1 appears in the captured image using a known method. The determination unit 17I also reads second attribute information associated with the terminal ID of the portable terminal 30B that transmitted the sensing result information acquired at the same acquisition time as the captured image in step S200 from the personal management information 18C (see FIG. 14 ), thereby deriving second attribute information for each second person P2 who is wearing the portable terminal 30B. The determination unit 17I determines whether there are first persons P1 and second persons P2 whose derived first attribute information and second attribute information match.

[0299] If the determination in step S214 is affirmative (step S214: Yes), the process proceeds to step S216. In step S216, if the first attribute information of the first person P1 derived based on the captured image matches the second attribute information of the second person P2 who is the wearer of the portable terminal 30B and that is previously stored in association with the second person ID of the wearer, the association unit 17E associates the first information of the first person P1 whose first attribute information matches with the second information of the second person P2 whose second attribute information matches (step S216). The association unit 17E associates the first information of the first person P1 whose first attribute information matches with the second person ID of the second person P2 whose second attribute information matches in the association management information 14D (see FIG. 7 ). The process then proceeds to step S226, which will be described later.

[0300] On the other hand, if the determination in step S214 is negative (step S214: No), the process proceeds to step S218. In step S218, the derivation unit 15C derives a first estimated motion of the first person P1 appearing in the captured image acquired in step S200 and captured by the image capture device 20 identified in step S204 (step S218). The derivation unit 15C also derives a second estimated motion of the second person P2 based on the sensing result information acquired in step S200 at the same acquisition time as the captured image (step S220).

[0301] The derivation unit 15C then registers the first estimated motion estimated for each first person P1, together with the first analysis result information used to derive the first estimated motion, in the photography management information 18A in association with the first person ID of the first person P1 and the captured image ID of the captured image used to derive the first analysis result information (see FIG. 12 ). The derivation unit 15C also registers the second estimated motion derived for each second person P2 in association with the sensing result information and second analysis result information used to derive the second estimated motion in the wearing management information 18B (see FIG. 13 ).

[0302] Next, the first estimated motion of the first person P1 derived in step S218 is compared with the second estimated motion of the second person P2 derived in step S220. Then, the comparison unit 15D determines whether the first estimated motion and the second estimated motion match and whether the first person P1 and the second person P2 exist (step S222).

[0303] If the determination in step S222 is negative (step S222: No), the process proceeds to step S236. If the determination in step S222 is positive (step S222: Yes), the process proceeds to step S224. In step S224, the association unit 17E associates the first person ID of the first person P1, whose first inferred action and the second inferred action match, with the second person ID of the second person P2, and registers them in the association management information 14D, thereby associating the first information about the first person P1 and the second information about the second person P2, whose inferred actions match (step S224). Then, the process proceeds to step S226.

[0304] In step S226, the derivation unit 17J analyzes the captured image acquired in step S200 and captured by the photographing device 20 identified in step S204 using a known method to derive at least one of the emotional information and interest level information of the first person P1 appearing in the captured image (step S226).

[0305] Then, the derivation unit 17J registers at least one of the derived emotion information and interest level information of the first person P1 in the photography management information 18A in association with the first person ID of the first person P1 (step S228) (see FIG. 12 ). Therefore, as shown in FIG. 12 , at least one of the emotion information and interest level information of the first person P1 derived from the photographed image is registered as first information in the photography management information 18A.

[0306] In addition, the derivation unit 17J derives at least one of the emotional information and interest level information of the second person P2 who is wearing the mobile terminal 30B using a known method based on the sensing result information acquired in step S200 and acquired at the same acquisition time as the captured image captured by the imaging device 20 identified in step S204 (step S230).

[0307] Then, the derivation unit 17J registers at least one of the emotion information and the interest level information of the second person P2 derived in step S230 in the wearing management information 18B in association with the second person ID of the second person P2 (step S232) (see FIG. 13 ). Therefore, as shown in FIG. 13 , at least one of the emotion information and the interest level information of the second person P2 derived based on the information obtained from the mobile terminal 30B is registered as second information in the wearing management information 18B.

[0308] Next, the output control unit 15F outputs multimodal information in which the first information and the second information are associated with each other to the output device 16. The output control unit 15F uses, for example, the association management information 14D as the multimodal information. In this case, the output control unit 15F outputs, to the output device 16, the association management information 14D in which the first person ID, the first information, the second person ID, and the second information are associated with each other.

[0309] As described above, the first information of the first person P1 includes information related to a plurality of captured images of the first person P1 that differ in at least one of acquisition time and capture device 20. For example, the first information includes a captured image identified by a captured image ID, a capture device 20 that captured the captured image identified by a capture device ID, an acquisition time, first analysis result information including first attribute information, a first estimated action including a first movement trajectory, and at least one of emotion information and interest level information.

[0310] The second information of the second person P2 includes information about a plurality of pieces of sensing result information of the second person P2 obtained at different times. The second information of the second person P2 may also include detailed person information of the second person P2. For example, the second information includes sensing result information, second analysis result information, second estimated motion, second movement trajectory, detailed person information including second attribute information, and at least one of emotion information and interest level information of the second person P2.

[0311] Therefore, the output control unit 15F can output the multimodal information for each person P by outputting the multimodal information to the output device 16.

[0312] Next, the processing unit 17 determines whether to end the process (step S236). For example, the processing unit 17 executes the determination of step S236 by determining whether an instruction signal indicating the end of the process has been received from the input unit 12. If the determination in step S236 is negative (step S236: No), the processing unit 17 sets the acquisition time of the captured image, sensing result information, and location information of the mobile terminal 30 to be acquired by the acquisition unit 17A next to the time of the previous acquisition, and then returns to step S200. If the determination in step S236 is positive (step S236: Yes), this routine ends.

[0313] As described above, in the information processing system 10B of this embodiment, the imaging device 20 that captures an image in which the wearer of the portable terminal 30B appears is identified based on the position information of the portable terminal 30B. Then, in the information processing system 10B, the first information and the second information are associated with each other based on the first inferred motion and the second inferred motion derived based on the image captured by the identified imaging device 20 and the sensing result.

[0314] In this way, in the information processing system 10B of this embodiment, the first information and the second information are associated using a captured image taken by the imaging device 20 that captures an image in which the wearer of the mobile terminal 30B is captured.

[0315] Therefore, in the information processing system 10B of this embodiment, in addition to the effects of the above-described embodiment, multiple types of information related to the person P can be more efficiently integrated and managed.

[0316] In addition, in the information processing system 10B of this embodiment, location information of the mobile terminal 30B calculated by the mobile terminal 30B based on location information of one or more communication devices 70 that emit wireless signals and wireless signals received from the communication devices 70 is acquired.

[0317] Therefore, in the information processing system 10B of this embodiment, it is possible to identify the photographing device 20 that captures the image in which the wearer of the portable terminal 30B appears, using the position information of the portable terminal 30B calculated with high accuracy.

[0318] Furthermore, in the information processing system 10B of this embodiment, if the first attribute information of the first person P1 derived based on the captured image matches the second attribute information of the wearer of the portable terminal 30B that is stored in advance in association with the wearer's identification information, the information processing system 10B associates the first information with the second information. If the first attribute information and the second attribute information do not match, the information processing system 10B associates the first information with the second information based on a comparison result of the first inferred motion and the second inferred motion derived based on the captured image and the sensing result, respectively.

[0319] In this way, in the information processing system 10B of this embodiment, it is first determined whether the attribute information matches, and if the attribute information does not match, the first information and the second information are associated based on the comparison result of the first estimated action and the second assumed action.

[0320] Therefore, in the information processing system 10B of this embodiment, in addition to the effects of the above-described embodiment, multiple types of information related to the person P can be more efficiently integrated and managed.

[0321] The first estimated motion includes a first movement trajectory of the first person P1 who appears in the captured image, and the second estimated motion includes a second movement trajectory represented by the transition of the position information of the mobile device 30 B. Therefore, the information processing system 10B of this embodiment can provide multimodal information that further includes a movement trajectory.

[0322] The first information includes at least one of emotion information and interest level information of the first person P1, and the second information includes at least one of emotion information and interest level information of the second person P2.

[0323] Therefore, the information processing system 10B of this embodiment can provide multimodal information that further includes at least one of emotion information and interest level information of the person P.

[0324] The multimodal information generated by the information processing system 10B of this embodiment is applicable to various systems.

[0325] For example, the multimodal information generated by the information processing system 10B of this embodiment can be used for marketing analysis to provide products and services to person P. Furthermore, the results of this marketing analysis can also be used for effective review processes such as effective store placement within the real-space RS, store layout, and events to be held within the real-space RS. Furthermore, the multimodal information generated by the information processing system 10B can also be used to track the behavior of person P. Specifically, for example, an information processing device such as a server device may perform processing such as managing which of multiple checkpoints in a predetermined real-space RS each person P has passed through based on the multimodal information and electronically granting electronic money or the like to person P according to the number of checkpoints passed through. This processing can also encourage person P to travel along a route or location desired by an administrator within the real-space RS, or to purchase a specific product.

[0326] Next, an example of the hardware configuration of the information processing system 10, the mobile terminal 30, the information processing system 10B, and the mobile terminal 30B of the above embodiment will be described.

[0327] FIG. 17 is a diagram illustrating an example of a hardware configuration of the information processing system 10, the mobile terminal 30, the information processing system 10B, and the mobile terminal 30B according to the above embodiment.

[0328] The information processing system 10, the mobile terminal 30, the information processing system 10B, and the mobile terminal 30B of the above embodiments have a hardware configuration that utilizes a conventional computer, in which a CPU (Central Processing Unit) 80, a ROM (Read Only Memory) 82, a RAM (Random Access Memory) 84, an I / F 86, etc. are interconnected by a bus 88.

[0329] The CPU 80 is a computing device that controls the information processing system 10, the mobile terminal 30, the information processing system 10B, and the mobile terminal 30B of the above-described embodiments. The ROM 82 stores programs and the like that realize information processing by the CPU 80. The RAM 84 stores data necessary for various processes by the CPU 80. The I / F 86 is an interface that is connected to the storage unit, input unit, output unit, sensor, communication unit, and the like, and is used to send and receive data.

[0330] In the information processing system 10, the mobile terminal 30, the information processing system 10B, and the mobile terminal 30B of the above embodiments, the CPU 80 reads a program from the ROM 82 onto the RAM 84 and executes it, thereby realizing each of the above functional units on the computer.

[0331] The programs for executing the above processes executed by the information processing system 10, the mobile terminal 30, the information processing system 10B, and the mobile terminal 30B of the above embodiments may be stored in a hard disk drive (HDD). Also, the programs for executing the above processes executed by the information processing system 10, the imaging device 20, and the mobile terminal 30 of the above embodiments may be provided by being pre-installed in the ROM 82.

[0332] In addition, the programs for executing the above processes executed by the information processing system 10, mobile terminal 30, information processing system 10B, and mobile terminal 30B of the above embodiments may be stored in an installable or executable file format on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD (Digital Versatile Disk), or flexible disk (FD) and provided as a computer program product. Furthermore, the programs for executing the above information processes executed by the information processing system 10, mobile terminal 30, information processing system 10B, and mobile terminal 30B of the above embodiments may be stored on a computer connected to a network such as the Internet and provided by downloading via the network. Furthermore, the programs for executing the above information processes executed by the information processing system 10, mobile terminal 30, information processing system 10B, and mobile terminal 30B of the above embodiments may be provided or distributed via a network such as the Internet.

[0333] Although the above describes an embodiment, the embodiment is presented as an example and is not intended to limit the scope of the invention. This novel embodiment can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. This embodiment and its modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims.

[0334] The present technology may also be configured as follows: (1) An information processing system including at least one processor, wherein, when a first estimated motion of a first person appearing in a captured image matches a second estimated motion of a person wearing the portable device derived based on sensing results by a sensor mounted on the portable device, the at least one processor outputs to an output device multimodal information associating first information about the first person in the captured image with second information acquired from the portable device. (2) The information processing system described in (1), wherein the at least one processor identifies a camera that captured the captured image in which the person wearing the portable device appears, based on location information of the portable device, and associates the first information with the second information based on the first estimated motion and the second estimated motion derived based on the captured image captured by the identified camera and the sensing results. (3) The information processing system according to (2), wherein the at least one processor acquires location information of one or more communication devices that emit wireless signals and the location information of the portable device calculated by the portable device based on the wireless signals received from the communication devices. (4) The at least one processor associates the first attribute information with the second information when first attribute information of the first person derived based on the captured image matches second attribute information of the wearer of the portable device that is stored in advance in association with identification information of the wearer of the portable device, and associates the first information with the second information when the first attribute information and the second attribute information do not match. (5) The information processing system according to any one of (1) to (4), wherein the first estimated motion includes a first movement trajectory of the first person appearing in the captured image, and the second estimated motion includes a second movement trajectory represented by a transition in location information of the mobile device.(6) The information processing system according to any one of (1) to (5), wherein the first information includes first analysis result information of the first person appearing in the captured image, and the second information includes second analysis result information of data from the sensor. (7) The information processing system according to any one of (1) to (6), wherein the first information includes at least one of emotion information and interest level information of the first person. (8) The information processing system according to any one of (1) to (7), wherein the second information includes at least one of emotion information and interest level information of the wearer. (9) The information processing system according to any one of (1) to (8), wherein the at least one processor transmits, to the mobile device, action instruction information that prompts the mobile device to perform an action, and associates the first information with the second information based on a comparison result of the first estimated action and the second estimated action derived based on the captured image and the sensing result obtained after the transmission of the action instruction information. (10) The information processing system according to (9), wherein, when a wearer related to the second information not associated with the first information is present when processing is performed for a predetermined period, the at least one processor transmits the action instruction information to the portable terminal worn by the wearer. (11) The information processing system according to any one of (1) to (10), wherein the sensor includes at least one of a first sensor mounted on the portable terminal worn on an arm of the wearer and a second sensor that detects information related to the soles of the wearer's feet. (12) The information processing system according to any one of (1) to (11), wherein the at least one processor derives the first estimated action of the first person based on skeletal information of the first person appearing in the captured image. (13) The information processing system according to any one of (1) to (12), wherein the at least one processor continues deriving the first estimated motion of one or more first persons appearing in the captured image, the first information relating to the first person not being associated with the second information.(14) The information processing system according to any one of (1) to (13), wherein, when a new first person not appearing in the captured images previously acquired is identified, the at least one processor compares the first estimated motion of the new first person with the second information about the wearer not associated with the first information. (15) The information processing system according to any one of (1) to (14), wherein the at least one processor identifies the same first person appearing in a plurality of the captured images captured by at least one of different image capturing devices and at different capture times, and associates the first information about the identified first person with the second information about the wearer. (16) An information processing method executed by an information processing system, comprising: when a first estimated motion of a first person appearing in a captured image matches a second estimated motion of a person wearing the portable device derived based on a sensing result by a sensor mounted on the portable device, outputting to an output device multimodal information associating first information about the first person in the captured image with second information acquired from the portable device. (17) An information processing program to be executed by a computer, comprising: when a first estimated motion of a first person appearing in a captured image matches a second estimated motion of the person wearing the portable device derived based on a sensing result by a sensor mounted on the portable device, outputting to an output device multimodal information associating first information about the first person in the captured image with second information acquired from the portable device.

[0335] 10, 10B Information processing system 15, 17 Processing unit 20 Imaging device 30, 30B Portable terminal 34A First sensor 34B Second sensor 70 Communication device

Claims

1. An information processing system comprising at least one processor, wherein, when a first estimated motion of a first person appearing in a captured image matches a second estimated motion of a person wearing the mobile device derived based on sensing results by a sensor mounted on the mobile device, the at least one processor outputs to an output device multimodal information that associates first information about the first person in the captured image with second information obtained from the mobile device.

2. The information processing system of claim 1, wherein the at least one processor identifies a photographing device that captures the image in which the wearer of the portable device appears based on location information of the portable device, and associates the first information with the second information based on the first estimated motion and the second estimated motion derived based on each of the image captured by the identified photographing device and the sensing result.

3. The information processing system according to claim 2, wherein the at least one processor acquires location information of one or more communication devices that emit wireless signals, and the location information of the mobile terminal calculated by the mobile terminal based on the wireless signals received from the communication devices.

4. The information processing system of claim 2, wherein the at least one processor, when first attribute information of the first person derived based on the captured image matches second attribute information of the wearer stored in advance in association with identification information of the wearer of the mobile terminal, associates the first information with the second information; and when the first attribute information and the second attribute information do not match, associates the first information with the second information based on a comparison result of the first estimated motion and the second estimated motion derived based on the captured image and the sensing result, respectively.

5. The information processing system of claim 1, wherein the first estimated motion includes a first movement trajectory of the first person appearing in the captured image, and the second estimated motion includes a second movement trajectory represented by a transition in location information of the mobile device.

6. An information processing system as described in claim 1, wherein the first information includes first analysis result information of the first person appearing in the captured image, and the second information includes second analysis result information of the data from the sensor.

7. The information processing system according to claim 1, wherein the first information includes at least one of emotional information and interest level information of the first person.

8. The information processing system according to claim 1, wherein the second information includes at least one of emotional information and interest level information of the wearer.

9. The information processing system of claim 1, wherein the at least one processor transmits to the mobile terminal action instruction information that prompts an action, and associates the first information with the second information based on a comparison result of the first estimated action and the second estimated action derived based on each of the captured image and the sensing result acquired after the transmission of the action instruction information.

10. The information processing system of claim 9, wherein the at least one processor, when executing processing for a predetermined period, if there is a wearer related to the second information that is not associated with the first information, transmits the operation instruction information to the portable terminal worn by the wearer.

11. The information processing system according to claim 1, wherein the sensor includes at least one of a first sensor mounted on the mobile terminal worn on the wearer's arm, and a second sensor that detects information relating to the soles of the wearer's legs.

12. The information processing system according to claim 1, wherein the at least one processor derives the first estimated motion of the first person based on skeletal information of the first person appearing in the captured image.

13. The information processing system of claim 1, wherein the at least one processor continues to derive the first estimated motion of one or more first persons appearing in the captured image, for which the second information is not associated with the first information regarding the first person.

14. The information processing system of claim 1, wherein, when a new first person who does not appear in the previously acquired captured image is identified, the at least one processor compares the first estimated movement of the first person with the second information about the wearer and the second estimated movement of the wearer not associated with the first information.

15. The information processing system of claim 1, wherein the at least one processor identifies the same first person appearing in multiple captured images taken using at least one of different image capture devices and different capture times, and associates the first information about the identified first person with the second information about the wearer.

16. An information processing method executed by an information processing system, comprising: when a first estimated movement of a first person appearing in a captured image matches a second estimated movement of a person wearing the mobile device derived based on sensing results from a sensor mounted on the mobile device, outputting to an output device multimodal information that associates first information about the first person in the captured image with second information obtained from the mobile device.

17. An information processing program to be executed by a computer, which, when a first estimated movement of a first person appearing in a captured image matches a second estimated movement of a person wearing the mobile device derived based on sensing results from a sensor mounted on the mobile device, outputs to an output device multimodal information that associates first information about the first person in the captured image with second information obtained from the mobile device.

Images