System for identifying user in image

Abstract

This system stores position management information for managing position related information indicating values associated with each position in an image. The system: acquires an image captured by an imaging device; identifies a person in the image by means of image recognition processing; receives position related information measured by a portable device of a user from the portable device; refers to the position management information to identify a time variation in the position related information associated with movement of the person in the image; and identifies, in the image, the user holding the portable device on the basis of a degree of similarity between the time variation in the position related information received from the portable device and the time variation in the position related information of the person in the image.

Description

System for identifying users in images Incorporation by Reference

[0001] This application claims priority from Japanese Patent Application No. 2023-122855, filed on July 27, 2023, the contents of which are incorporated herein by reference.

[0002] The present invention relates to a technique for identifying a user in an image.

[0003] As user needs become more diverse, there is a demand for providing services tailored to each user in various situations. For example, it may be necessary to provide services tailored to each user at the entrance of an apartment building, a theme park, etc. By acquiring digital attribute information indicating what contracts and attributes a viewed user has and making the digital attribute information visible to a service provider, the service provider can quickly provide services tailored to each user.

[0004] For example, Patent Literature 1 discloses a method for identifying a user photographed by a camera using facial recognition. However, biometric information such as facial recognition is personal information, and in recent years, management of personal information has become stricter and awareness of privacy has increased year by year. Therefore, methods using personal information such as facial recognition increase information management costs and barriers to user subscription to services.

[0005] Japanese Patent Application Laid-Open No. 2016-122300

[0006] Therefore, there is a need for a technology that can identify a user shown in an image without using personal information about the user, such as biometric information.

[0007] One aspect of the present invention is a system for identifying a user in an image, comprising one or more processors and one or more storage devices, wherein the one or more storage devices store location management information that manages location-related information indicating values ​​associated with each position in the image, and the one or more processors acquire an image captured by an imaging device, identify a person in the image through image recognition processing, receive from a user's portable device location-related information measured by the portable device, and identify, by referring to the location management information, a time variation in the location-related information associated with the movement of the person in the image, and identify the user holding the portable device in the image based on the similarity between the time variation in the location-related information received from the portable device and the time variation in the location-related information of the person in the image.

[0008] According to one aspect of the present invention, it is possible to identify a user shown in an image without using personal information of the user such as biometric information.

[0009] 1A and 1B are schematic diagrams illustrating an example of operation of a digital attribute information linkage system according to an embodiment of the present specification. A configuration example of a digital attribute information presentation device is shown. A configuration example of a location-related information table is shown. A configuration example of a digital attribute information table is shown. A diagram illustrating the processing of each program of the digital attribute information presentation device. A flowchart of an example of processing by an image processing unit is shown. A diagram illustrating the configuration and processing of a location-related information processing unit. A flowchart of an example of processing by a location-related information variation acquisition unit is shown. A flowchart of an example of processing by a user location estimation unit is shown. An example of time variation of three pieces of location-related information is shown. A flowchart of an example of processing by a digital attribute information visualization unit is shown. An example of an area within the camera's imaging area where location-related information has been acquired and other areas where location-related information has not been acquired is shown. A user's movement route is shown in an imaging area that is composed of areas where location-related information has been acquired and other areas. A flowchart of an example of processing by a location-related information table update unit is shown. A state in which only one person, the user, is present within the camera's imaging area is shown. An example of an area where the received radio wave strength from a beacon exceeds a predetermined threshold is shown. An area within the camera's imaging area where location-related information has been acquired and a person's movement route are shown.

[0010] Hereinafter, several embodiments of the present invention will be described with reference to the drawings. For convenience, the following embodiments will be divided into multiple sections or embodiments. However, unless otherwise specified, these are not unrelated and one is related to the other in terms of partial or complete modifications, details, supplementary explanations, etc. Each embodiment may be implemented individually or in combination.

[0011] Furthermore, in the following examples, when referring to the number of elements (including the number, numerical value, amount, range, etc.), unless otherwise specified or when it is clearly limited to a specific number in principle, it is not limited to that specific number and may be more or less than the specific number.

[0012] Furthermore, it goes without saying that in the following examples, the components (including element steps, etc.) are not necessarily essential unless otherwise specified or unless they are clearly considered essential in principle.

[0013] Similarly, in the following examples, when referring to the shapes, positional relationships, etc. of components, etc., unless otherwise specified or clearly considered otherwise in principle, it is intended to include those that are substantially similar or approximate to those shapes, etc. The same applies to numerical values ​​and ranges.

[0014] In the following explanation, expressions such as "xxx table" may be used, but the information may be data of any structure. In the following explanation, the structure of each table is an example, and one table may be divided into two or more tables, or two or more tables may all or partly be one table.

[0015] The processor realizes a predetermined function by executing a program stored in a primary storage device. The primary storage device stores the program executed by the processor and data necessary for the program execution. The program includes an operating system (OS) (not shown) as well as other programs. The processor may include multiple chips and multiple packages.

[0016] A program is executed by a processor to perform a predetermined process using a storage device and a communication port (communication device). Therefore, in this embodiment and other embodiments, a description using the program as the subject may also be a description using the processor as the subject. Alternatively, the process executed by a program is a process performed by the computer and computer system on which the program runs.

[0017] A processor operates according to a program, thereby acting as a functional unit (means) that realizes a predetermined function. For example, a processor functions as a unit (means) by operating according to a certain program, and as a unit (means) by operating according to a certain program. The same is true for other programs. Furthermore, a processor also operates as a functional unit (means) that realizes each of the multiple processes executed by each program. A computer and a computer system are devices and systems that include these functional units (means).

[0018] 1 is a schematic diagram illustrating an example of operation of a digital attribute information linkage system according to an embodiment of the present specification. The digital attribute information linkage system detects a user in an image and acquires digital attribute information of the detected user. The digital attribute information linkage system associates the user with the user's digital attribute information in the image and presents the associated information to a service provider.

[0019] Digital attribute information is digitized attribute information about a user that cannot be known from the user's appearance. Examples of the service attribute information that can be presented include residence information and visiting destination information for apartments and office buildings in security support, theme park membership information (e.g., fast passes) in customer service support, and boarding station information in railway station entrance / exit management for gateless ticket gates.

[0020] In the configuration example shown in Fig. 1, the digital attribute information linkage system includes a camera 11 which is an imaging device, a beacon device 12, a communication base station 13, and a digital attribute information presentation device 30. In the example shown in Fig. 1, the orientation and angle of view of the camera 11 are fixed. In other words, the area captured by the camera 11 is fixed. The camera 11 captures, for example, a moving image of a specific indoor space in a building. Note that the moving image is composed of successive frame images (still images).

[0021] 1, a plurality of users are present in a room including the imaging area 113 of the camera 11. Specifically, three users 15A, 15B, and 15C are present. These three users 15A, 15B, and 15C are present within the imaging area 113 of the camera 11. The three users 15A, 15B, and 15C are carrying mobile devices 16A, 16B, and 16C, respectively.

[0022] An example of the mobile device is a smartphone. The mobile device is equipped with a program for receiving services from the digital attribute information linkage system. In the configuration example shown in FIG. 1, the mobile devices 16A and 16B measure the radio wave intensity received from the beacon device 12, and then transmit the measured radio wave intensity and the corresponding user's digital attribute information to the digital attribute information presentation device 30 via the communication base station 13. The mobile device 16C is not eligible for this service.

[0023] Radio wave strength depends on the distance from the fixed beacon device 12 and is an example of location-related information that indicates the location of the mobile device, i.e., the corresponding user. Another example of location-related information is geomagnetic field strength. At different locations, geomagnetic field strength may show different values. The mobile device can measure geomagnetic field strength using a program installed in it. This is particularly useful indoors where it is difficult to obtain a GPS (Global Positioning System) signal.

[0024] In the example described below, the digital attribute information presentation device 30 receives user attribute information to be presented to the service provider from the user's mobile device. This facilitates the collection and management of user attribute information to be presented to the service provider. In another example, the mobile device may not transmit the digital attribute information. For example, the digital attribute information presentation device 30 may use the user ID received together with the location-related information as a key to search a database of the user's digital attribute information and obtain the corresponding digital attribute information. Note that the mobile device may be another type of device or may be composed of multiple separate components.

[0025] As described above, the mobile devices 16A and 16B transmit the user's location-related information and digital attribute information to the communication base station 13 via a predetermined protocol, such as Wi-Fi, Bluetooth (registered trademark), or cellular. The transmitted information includes an ID that uniquely identifies the user. The communication base station 13 transfers the received information to the digital attribute information presentation device 30 via the network.

[0026] The digital attribute information presentation device 30 receives digital attribute information and position-related information of each user from the communication base station 13, and receives image data from the camera 11. The digital attribute information presentation device 30 identifies position-related information that matches the user (user position) in the image from the position of each user in the image (within the angle of view) and the position-related information of each user.

[0027] As will be described later, the digital attribute information presentation device 30 holds management information indicating the relationship between a position in an image and position-related information, and can obtain position-related information corresponding to a position in an image by referring to the management information. By comparing the position-related information corresponding to a position in an image with the position-related information received from a user, the received position-related information can be matched with the user in the image. This makes it possible to match (associate) the user in the image with the user's digital attribute information.

[0028] The digital attribute information presentation device 30 displays the image received from the camera 11 together with the digital attribute information received from the mobile device on its display device or on the display device 20 of a terminal connected via a network, thereby visualizing the user's digital attribute information. In Figure 1, user images 21A, 21B, and 21C are images of users 15A, 15B, and 15C, respectively. Furthermore, user images 21A and 21B are surrounded by bounding boxes 22A and 22B, respectively, generated by object detection processing.

[0029] The display device 20 displays digital attribute information 23A in association with a user image 21A. It also displays digital attribute information 23B in association with a user image 21B. No digital attribute information is displayed for a user image 21C that is not a service target. The service provider can visually check the digital attribute information of the service user captured by the camera 11 on the display device 20 and quickly take any necessary action.

[0030] Although FIG. 1 and the examples described below use a camera as an example of an imaging device that captures an image of a user, other types of imaging devices, such as Lidar devices, that can generate images that allow a user to be detected through image recognition processing that performs object detection may also be used.

[0031] The portable device may transmit necessary information to the communication base station 13 only when it is within a preset range. For example, the information may be transmitted when the radio wave reception strength exceeds a threshold, that is, when the distance to the beacon device 12 is within a range closer than the threshold. By placing the beacon device 12 near the camera 11, unnecessary processing of the portable device that is not captured by the camera 11 can be reduced. Alternatively, for example, the portable device may use GPS to transmit information only while the user is inside the building and only while it can receive Bluetooth radio waves or Wi-Fi radio waves from a device installed near the camera.

[0032] 2 shows an example of the configuration of the digital attribute information presentation device 30. The digital attribute information presentation device 30 includes an input device 310, an output device 314, a calculation device 311, a memory 312, an auxiliary storage device 313, and a communication interface (I / F) 315. The input device 310 includes a human interface such as a mouse and a keyboard. The output device 314 includes a display or a printer that outputs the calculation results of the digital attribute information presentation device 30. The communication interface 315 mediates communication with other devices such as the camera 11 and the communication base station 13.

[0033] The auxiliary storage device 313 stores various programs for implementing analysis processing by the digital attribute information presentation device 30, and the execution results of the processing, etc. The auxiliary storage device 313 stores a position relation information table 331 and a digital attribute information table 333.

[0034] Various programs and data stored in the auxiliary storage device 313 are deployed in the memory 312. In the example of Fig. 2, the memory 312 stores processing programs for the image processing unit 301, the user position estimation unit 302, the position-related information processing unit 303, and the digital attribute information visualization unit 304. The arithmetic unit 311 executes the programs loaded in the memory 312 and operates as the corresponding functional units, and is composed of, for example, one or more CPUs, GPUs, etc. The processing and calculations described below are executed by the arithmetic unit 311.

[0035] FIG. 3 shows an example of the configuration of the positional relation information table 331. The positional relation information table 331 is management information that indicates the relationship between positional relation information and positions within an image (image capture area). FIG. 3 shows received radio wave intensity as the positional relation information. As described above, the positional relation information may be, for example, geomagnetic intensity. The positional relation information table 331 divides the field of view (image capture area) of the camera 11 into multiple sections in a matrix and stores positional relation information for each section. Each point within each section is represented by a single position. The size and shape of the sections are arbitrary and may be the same or different.

[0036] In the example configuration shown in Fig. 3, row 401 indicates the X coordinate value of a divided section. Column 402 indicates the Y coordinate value of a divided section. The X axis is the horizontal axis of the image, and the Y axis is the vertical axis of the image, and they are perpendicular to each other. The positional relation information table 331 indicates the radio wave reception strength as the positional relation information for each divided section identified by the X coordinate and Y coordinate.

[0037] The positional relationship information for each section may be a single representative value or may indicate a probability distribution (for example, the mean and variance of a normal distribution). The heat map data indicated by the positional relationship information table 331, which is management information, may be expressed as a function rather than in a table format.

[0038] FIG. 4 shows an example of the configuration of the digital attribute information table 333. The digital attribute information table 333 stores and manages digital attribute information of users. In the example configuration shown in FIG. 4, the digital attribute information table 333 includes a user ID column 411 and multiple attribute columns 412. In FIG. 4, as an example, one attribute column indicating digital attribute A is indicated by reference numeral 412. The user ID uniquely identifies a user. One or multiple attributes of each user are managed.

[0039] 5 is a diagram illustrating the processing of each program of the digital attribute information presentation device 30. The image processing unit 301 acquires image data from the camera 11 and executes predetermined processing including person detection and identification of the person's movement route. The processing result by the image processing unit 301 is sent to the position-related information processing unit 303 and the user position estimation unit 302.

[0040] The positional relation information processing unit 303 updates (including storing initial data) the positional relation information table 331 based on the processing result from the image processing unit 301 and the positional relation information acquired from the user. Furthermore, the positional relation information processing unit 303 identifies fluctuations in the positional relation information (e.g., radio wave intensity) of each person in the image based on the processing result from the image processing unit 301.

[0041] The user position estimation unit 302 compares the time variation of the position-related information (e.g., radio wave intensity) acquired from each user with the time variation of the position-related information of each person in the image acquired from the position-related information processing unit 303, and associates the people in the image with the user (user ID).

[0042] The digital attribute information visualization unit 304 generates an image that associates the user with the digital attribute information, and presents it to the service provider.

[0043] The processing by each program is described in detail below. FIG. 6 shows a flowchart of an example of processing by the image processing unit 301. The image processing unit 301 acquires and receives image data captured by the camera 11 (S11). The image processing unit 301 performs image processing on each image to detect people (S12). The detection of people within an image may use a known image recognition technology, such as a machine learning technique such as YOLO (You Only Look Once) or SSD (Single Shot Multibox Detector), or a technique other than machine learning. The image processing unit 301 tracks each person detected in successive images and identifies the movement path of each person within the images (S13). Each movement path may be assigned a unique ID.

[0044] 7 is a diagram illustrating the configuration and processing of the positional relation information processing unit 303. The positional relation information processing unit 303 includes a positional relation information table updating unit 336 and a positional relation information variation acquisition unit 338. The positional relation information table updating unit 336 updates the information held in the positional relation information table 331. Details of the processing of the positional relation information table updating unit 336 will be described later.

[0045] The positional relation information variation acquisition unit 338 identifies the time variation of the positional relation information of each person based on the movement paths of each person in the image acquired from the image processing unit 301 and the positional relation information table 331 .

[0046] 8 is a flowchart showing an example of processing by the positional relation information variation acquisition unit 338. The positional relation information variation acquisition unit 338 acquires information on the movement routes of all users detected in the image from the image processing unit 301. The positional relation information variation acquisition unit 338 acquires positional relation information corresponding to each of the movement routes of all users from the positional relation information table 331 (S21).

[0047] Specifically, the position-related information variation acquisition unit 338 identifies the coordinates of the divided sections in the user's movement route (changes in the X and Y coordinate values) and acquires the radio wave strength (position-related information) corresponding to each of the coordinates, thereby acquiring an array indicating the variation in radio wave strength in response to changes in the coordinates within the image.

[0048] Next, the position-related information variation acquisition unit 338 acquires time variations in the position-related information of all users (S22). Specifically, the position-related information variation acquisition unit 338 acquires combinations of the coordinates and time of each point on the movement route from the time of the image, and acquires time variations in the radio wave strength (position-related information) corresponding to each movement route from the relationship between the coordinates and radio wave strength acquired in step S21. The time variations in radio wave strength indicate fluctuations in radio wave strength over time and are an array of combinations of time and radio wave strength.

[0049] 9 shows a flowchart of an example of processing by the user position estimation unit 302. First, the user position estimation unit 302 determines whether the data storage rate of the positional relation information table 331 is less than a threshold (S31). If the data storage rate is less than the threshold (S31: YES), this means that the positional relation information table 331 does not store enough positional relation information (e.g., radio wave intensity values) to identify the user's coordinates within the image. Processing is then executed to update the positional relation information table 331.

[0050] The user position estimation unit 302 determines whether position-related information has been received from only one user and whether the number of people in the image (angle of view) is one (S32). If this condition is not met (S32: NO), this processing flow ends. If the above condition is met (S32: YES), the user position estimation unit 302 transmits the movement route of only one specific user in the image (angle of view) to the position-related information processing unit 303. This information is used to update the position-related information table 331, as will be described later.

[0051] If the data storage rate of the positional relation information table 331 is equal to or greater than the threshold value (S31: NO), the flow proceeds to step S34. The user position estimation unit 302 acquires time fluctuations in the positional relation information of all people in the image from the positional relation information processing unit 303 (S34).

[0052] Furthermore, the user position estimation unit 302 acquires time variation of the position-related information from each of all users (S35). The position-related information transmitted by a user includes time information and the user ID of the user. In other words, the position-related information from a user indicates the time when the position-related information (e.g., radio wave intensity) was acquired and the position-related information itself.

[0053] Next, the user position estimation unit 302 compares the time variation of the position related information received from each user with the time variation of the position related information of each person in the image received from the position related information processing unit 303, and searches for a matching pair (S36).

[0054] When the user position estimation unit 302 finds a time variation in the position-related information of a user that matches the time variation in the position-related information of the person in the image, it acquires the movement route of the person in the image (user) from the movement routes of all people acquired from the image processing unit 301, and transmits the movement route and the time variation in the position-related information of the user to the position-related information processing unit 303 (S37). This information can be used to update the position-related information table 331.

[0055] Furthermore, the user position estimation unit 302 associates the user ID with the movement route in the image (change in position in the image over time) acquired from the image processing unit 301, and transmits the result to the digital attribute information visualization unit 304 (S38). Note that the digital attribute information visualization unit 304 may receive a combination of the user ID and the movement route ID from the user position estimation unit 302, and acquire information on the movement route together with image data from the image processing unit 301.

[0056] As described above, in step S36, the user position estimation unit 302 performs matching between the time variation of the position-related information (e.g., radio wave intensity) of the person in the image and the time variation of the position-related information (e.g., radio wave intensity) received from the user. Any matching method can be used, and one example will be described below.

[0057] 10 shows an example of time variations of three pieces of location-related information. The horizontal axis of the graph represents time, and the vertical axis represents location-related information (e.g., radio wave intensity). Line 551 represents time variations of location-related information estimated from the movement path of a person in the image. Line 552 represents time variations of location-related information received from user A. Line 553 represents time variations of location-related information received from user B.

[0058] The user position estimation unit 302 calculates the similarity between the time variation of the position-related information estimated from the movement path of the person in the image and the time variation of the position-related information received from each user. The user whose time variation of the position-related information received from the user whose similarity exceeds a threshold and is the highest is determined to be the person in the image. The similarity may be calculated, for example, using the sum of squared distances between the time variations. A smaller sum of squared distances indicates a higher similarity. The similarity may also be calculated between time variations over a predetermined period of time (e.g., 2 seconds). Note that if there is a blank period (described later) in the position-related information for the person in the image, that period is excluded from the calculation.

[0059] 11 shows a flowchart of an example of processing by the digital attribute information visualization unit 304. The digital attribute information visualization unit 304 executes the processing shown in FIG. 11 for each user identified in the image. The digital attribute information visualization unit 304 acquires digital attribute information of the user ID (ID of a specific user) received from the user position estimation unit 302 from the digital attribute information table 333 (S51). As described above, the digital attribute information of a user is stored in the digital attribute information table 333 in association with the user ID.

[0060] Furthermore, the digital attribute information visualization unit 304 identifies the user on the travel route associated with the user ID received from the user position estimation unit 302 in the image from the camera 11, and visualizes the attribute information of the user by displaying it in the image (S52). This allows the service provider to instantly know the user and their attributes in the camera image. Note that the digital attribute information of the user obtained by associating the person in the image with the user may be used for purposes other than displaying it to the service provider.

[0061] An example of a method for updating the positional relation information table 331 will be described below. The initial state of the positional relation information table 331 indicates the X and Y coordinates of the partitions, but does not indicate any positional relation information corresponding to the coordinates. In order to identify a user in a camera image, it is necessary to store and update the positional relation information in the positional relation information table 331.

[0062] 12 shows an example of an area 601 for which positional relationship information has been acquired and other areas for which positional relationship information has not been acquired within the imaging area (angle of view) 600 of the camera 11. As will be described later, the positional relationship information to be registered in the positional relationship information table 331 can be generated from the user's movement path within the image (time-varying position) and the time-varying positional relationship information received from the user. The area 601 for which positional relationship information has been acquired is indicated by a dot pattern. In the positional relationship information table 331 shown in FIG. 3, the radio wave intensity for the area 601 for which positional relationship information has been acquired is stored, and the radio wave intensity for the other areas is left blank.

[0063] 13 shows a user's movement route 605 in an image capture area 600 that is configured with an area 601 for which positional relation information has been acquired and other areas in the image capture area 600. The movement route 605 passes through the area 601 for which positional relation information has been acquired and other areas for which positional relation information has not been acquired.

[0064] Graph 620 shows temporal fluctuations in positional relationship information that can be acquired from the positional relationship information table 331 according to the movement path 605. Periods 622 and 624 show periods during which the user moved within the area 601 for which positional relationship information has been acquired. Period 623 shows a period during which the user moved within an area for which positional relationship information has not been acquired. During periods 622 and 624, positional relationship information can be acquired from the positional relationship information table 331. On the other hand, during period 623, positional relationship information cannot be acquired from the positional relationship information table 331. In other words, during period 623, it is not possible to compare the positional relationship information received from the user with the positional relationship information determined from the movement path within the image and the positional relationship information table 331.

[0065] 14 is a flowchart illustrating an example of processing by the positional relation information table updating unit 336. When a user is identified in an image, the positional relation information table updating unit 336 fills in blank cells in the positional relation information table 331 based on the user's movement route in the image and the positional relation information received from the user.

[0066] The positional relation information table update unit 336 acquires positional relation information corresponding to the user's movement route identified in the image from the positional relation information table 331 (S61). The movement route indicates a change in a position within the image over time, and the change in positional relation information corresponding to the movement route over time can be obtained from the positional relation information table 331. If positional relation information corresponding to a position within the movement route is not registered in the positional relation information table 331, a value indicating this, such as a NULL value, is obtained.

[0067] The positional relation information table update unit 336 compares the time variation of the positional relation information received from the user with the time variation of the positional relation information obtained from the positional relation information table 331, and updates the positional relation information table 331 (S62). Specifically, the positional relation information of the divided sections for which positional relation information is not registered in the positional relation information table 331 is extracted from the positional relation information received from the user, and registered. Note that already registered positional relation information may be updated using new positional relation information received from the user.

[0068] In the above example, the position-related information of the divided section to which the user has moved after being identified is updated, but it is also possible to record the past movement route of the person in the image, identify which user the person is, and then update the values ​​of the blank areas on the past movement route in the position-related information table 331.

[0069] As described above, the location association information table 331 is updated based on the user's movement route identified in the image and the location association information received from the user. To associate a person detected in the image with a user, the location association information table 331 needs to include enough information to enable this (see, for example, step S31 by the user location estimation unit 302).

[0070] In one embodiment of this specification, the updating of the positional relation information table 331 by the positional relation information table update unit 336 includes two stages. In the first stage, new positional relation information is registered in the positional relation information table 331 when the registration rate of positional relation information is lower than a threshold (including 0). The first stage is executed, for example, before the start of normal operation of the system. In the next second stage, while the system is operating, information in the positional relation information table 331 is used to perform a process of identifying users in an image, and the positional relation information from the identified users is registered in an unregistered area of ​​the positional relation information table 331.

[0071] First, a method for registering the location-related information field in the first stage will be described. In the first stage, location-related information is acquired from a specific user in a state where the only person present within the imaging area (angle of view) of the camera 11 is the user 15. FIG. 15 shows a state where the only person present within the imaging area 113 of the camera 11 is the user 15. The user 15 is holding a portable device 16. The user 15 moves within the imaging area 113 of the camera 11 so as to fill as much of the entire imaging area 113 as possible. The user 15 can follow any route within the imaging area 113 of the camera 11.

[0072] Since only one person is captured in the image, the person detected in the image is user 15. The positional relation information table update unit 336 identifies the relationship between the position in the image and the positional relation information from the time variation of the positional relation information received from user 15 and the movement path (time variation of the position in the image) received from the image processing unit 301, and registers the positional relation information in the positional relation information table 331.

[0073] In this way, in the first stage, when the person transmitting the location related information is one specific user and one person is detected in the image, the location related information table update unit 336 registers new location related information in the location related information table 331. A method for making it more certain that the user transmitting the location related information is one specific user within the image capture area 113 is to include, within the image capture area 113, an area where the received radio wave strength exceeds a threshold.

[0074] 16 shows an example of an area 701 where the strength of radio waves received from the beacon device 12 exceeds a predetermined threshold. The area 701 is contained within the imaging area 113 of the camera 11. When the strength of radio waves received from the beacon device 12 exceeds the threshold, the program of the portable device 16 starts transmitting location-related information. After starting to transmit the location-related information, the portable device 16 continues to transmit the location-related information even if it leaves the area 701.

[0075] Next, a description will be given of a method for updating the positional relation information table 331 in the second stage. The positional relation information table update unit 336 refers to the positional relation information table 331, identifies a user in the image, and updates the positional relation information table 331 with the positional relation information received from the identified user.

[0076] 17 shows an area 601 where positional relation information has been acquired and a person's movement route 605 within the imaging area 600 of the camera 11. In the movement route 605, a portion 606 is within the area 601 where positional relation information has been acquired. Furthermore, a portion 607 is within an area where positional relation information has not been acquired. During the period of portion 606, as described above, the user (user ID) of the person on the movement route is identified by the user position estimation unit 302.

[0077] In part 607, it is not possible to know the time variation of the positional relationship information by referring to the positional relationship information table 331. However, the user of the travel route 605 has already been identified using the positional relationship information table 331 during the period of part 606. Therefore, it is known that the positional relationship information received from the specific user during the period of part 607 is the positional relationship information of each point in part 607.

[0078] The positional relation information table update unit 336 registers the positional relation information received from the specific user for portion 607 in the cell of the corresponding divided section in the positional relation information table 331. As a result, the positional relation information for area 608 is newly registered in the positional relation information table 331. By repeatedly identifying users in areas where positional relation information has been acquired and updating the positional relation information table 331 with positional relation information from specific users in areas where positional relation information has not been acquired, the positional relation information table update unit 336 can increase the amount of registered information in the positional relation information table 331 and improve the accuracy of the in-image user identification process.

[0079] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0080] Furthermore, the above-described configurations, functions, processing units, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. Furthermore, the above-described configurations, functions, etc. may be implemented in software by a processor interpreting and executing a program that implements each function. Information such as the program, table, and file that implements each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card or SD card.

[0081] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

Claims

1. A system for identifying a user in an image, comprising: one or more processors; and one or more storage devices, wherein the one or more storage devices store location management information that manages location-related information indicating values ​​associated with each position in the image, wherein the one or more processors: acquire an image captured by an imaging device; identify a person in the image through image recognition processing; receive from a user's portable device location-related information measured by the portable device; identify a time variation in the location-related information associated with the movement of the person in the image by referring to the location management information; and identify a user holding the portable device in the image based on a similarity between the time variation of the location-related information received from the portable device and the time variation of the location-related information of the person in the image.

2. A system as described in claim 1, wherein the one or more processors update the location management information, and in a state where the first person is the only person within the imaging range of the imaging device, the updating comprises: identifying the first person in an image from the imaging device by image recognition processing; determining the relationship between each position on the movement path of the first person within the image and time; acquiring location related information from the first person's portable device; and associating each position on the movement path with the location related information from the first person's portable device at the same time for each position, and registering them in the location management information.

3. A system as described in claim 1, wherein the one or more processors update the location management information, the updating comprising: identifying a movement route of the identified user within the image; extracting, from the location related information received from the user's mobile device, location related information for positions on the movement route for which the location management information does not store location related information; and registering the extracted location related information in the location management information.

4. A system according to claim 1, wherein the determination of the similarity excludes location-related information of locations that are not registered in the location management information.

5. A system according to claim 1, wherein the one or more processors receive attribute information of the user from the mobile device of the user and register the attribute information in attribute management information.

6. A system according to claim 1, wherein the one or more processors display attribute information of the user on a display device together with an image captured by the imaging device.

7. A system as claimed in claim 2, wherein the location related information indicates radio wave strength received from a radio wave transmitter, and the entire range satisfying the conditions of radio wave strength measured by the mobile device is included within the imaging area of ​​the imaging device.

8. A method for a system to identify a user in an image, the system storing location management information that manages location-related information indicating values ​​associated with each location in the image, the method comprising the system acquiring an image captured by an imaging device, identifying a person in the image by image recognition processing, receiving from a user's portable device location-related information measured by the portable device, identifying a time variation in the location-related information associated with the movement of the person in the image by referring to the location management information, and identifying the user holding the portable device in the image based on a similarity between the time variation in the location-related information received from the portable device and the time variation in the location-related information of the person in the image.

Images