Location management system, location management method, and location management program

Abstract

To provide a location management system, a location management method, and a location management program for enhancing security.SOLUTION: A location management system A1 includes: a control unit 21 connected to a camera C1; and a location information storage unit 24. Then, the control unit 21 identifies a person image area in which a person is recognized in a captured image of the camera, sequentially identifies locations of the person according to positions of the person image area, and records the identified locations in the location information storage unit 24. When a code image is detected in the person image area, the control unit 21 records a code corresponding to a code image in the location information storage unit 24 in association with the location.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a location management system, a location management method, and a location management program for managing the location of a person in a specific area. [Background technology]

[0002] Conventionally, human movement is sometimes controlled in areas requiring security. For this reason, entrance / exit control devices have been considered to improve security (see, for example, Patent Document 1). The entrance / exit control device described in this patent document detects the number of persons to be determined who are entering or leaving a security area requiring security by image analysis of captured images. Furthermore, the number of persons to be determined who are entering or leaving the security area is detected by reading color barcodes attached to the persons to be determined. Then, if the number of people detected by image analysis and the number detected by the color barcode match, the person is permitted to enter or leave the security area.

[0003] Furthermore, an object tracking device that tracks an object included in a video is under study (see, for example, Patent Document 2). The object tracking device described in this patent document detects feature points in each frame by inputting each frame image of a video into a machine learning model that can output the type of subject. The object is tracked by comparing the positions of the feature points in each frame. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-173890 [Patent Document 2] Japanese Patent Application Publication No. 2019-185210 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology described in Patent Document 1 cannot manage the movement of people within a security area, and the technology described in Patent Document 2 cannot identify a tracked object. [Means for solving the problem]

[0006] A location management system that solves the above problem includes a camera and a control unit connected to a location information storage unit. The control unit identifies a person image area in which a person is recognized in an image captured by the camera, sequentially identifies the location of the person according to the position of the person image area, and records the location in the location information storage unit. When a code image is detected in the person image area, the control unit associates a code corresponding to the code image with the location and records the code in the location information storage unit. [Effects of the Invention]

[0007] According to the present invention, security can be improved by managing the location of users. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram of a location management system according to an embodiment. [Figure 2] FIG. 2 is an explanatory diagram of a hardware configuration of the embodiment. [Figure 3] FIG. 2 is an explanatory diagram of a processing procedure according to an embodiment. [Figure 4] FIG. 2 is an explanatory diagram of a processing procedure according to an embodiment. [Figure 5] FIG. 2 is an explanatory diagram of a captured image according to an embodiment. [Figure 6] FIG. 2 is an explanatory diagram of a captured image according to an embodiment. [Figure 7] FIG. 10 is an explanatory diagram of a processing procedure of a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, one embodiment of the location management system will be described with reference to FIGS. In this embodiment, a location management system, location management method, and location management program are described for use in managing the location of workers (users) within a factory. In this embodiment, a code image held by a user is recognized to identify the user, and image tracking technology is used to track the user's location within the managed area. Chameleon Code (registered trademark) is used as the code image. Chameleon Code is a code image that displays a code by arranging multiple colors (cyan, magenta, yellow, and black) in a predetermined layout. This Chameleon Code can be recognized collectively even when multiple code images are included in a single captured image. As shown in FIG. 1, the location management system A1 is configured with a camera C1, a management terminal 10, a management server 20, and the like, which are connected via a network.

[0010] (Hardware configuration description) 2, the hardware configuration of the information processing device H10 that constitutes the management terminal 10 and management server 20 will be described. The information processing device H10 includes a communication device H11, an input device H12, a display device H13, a storage device H14, and a processor H15. Note that this hardware configuration is an example, and it can also be realized by other hardware.

[0011] The communication device H11 is an interface that establishes a communication path with another device and executes data transmission and reception, and is, for example, a network interface or a wireless interface.

[0012] The input device H12 is a device that accepts input of various information, such as a mouse, a keyboard, etc. The display device H13 is a display or the like that displays various information. The storage device H14 is a storage device that stores data and various programs for executing various functions of the management terminal 10 and management server 20. Examples of the storage device H14 include a ROM, a RAM, and a hard disk.

[0013] The processor H15 uses programs and data stored in the storage device H14 to control each process in the management terminal 10 and management server 20. Examples of the processor H15 include a CPU, GPU, and MPU. The processor H15 loads programs stored in ROM or the like into RAM and executes various processes for each process.

[0014] The processor H15 is not limited to a processor that performs all of its processing using software. For example, the processor H15 may include a dedicated hardware circuit (e.g., an application-specific integrated circuit (ASIC)) that performs hardware processing for at least some of the processing it performs. The processor includes a CPU, a GPU, and memory such as RAM and ROM, which stores program code or instructions configured to cause the CPU and GPU to perform processing. Memory, i.e., computer-readable medium, includes any available medium that can be accessed by a general-purpose or dedicated computer.

[0015] (Explanation of location management system A1) The camera C1 transmits color video images of a predetermined field of view of the management target area to the management server 20. The camera C1 is installed in a position (for example, on the ceiling) in the factory where it can overlook the management target area from above. The camera C1 then generates captured images of the management target area below (predetermined capture range) from above.

[0016] The management terminal 10 is a computer terminal used by an administrator who manages the locations of users. The management server 20 is a computer that manages the location of users, and includes a control unit 21, a user information storage unit 22, an area information storage unit 23, and a location information storage unit 24.

[0017] The control unit 21 performs the processes described below (processing at the image recognition stage, tracking stage, and code recognition stage). By executing a management program for this purpose, the control unit 21 functions as an image recognition unit 211, a tracking unit 212, and a code recognition unit 213.

[0018] The image recognition unit 211 executes a process of recognizing a person in a captured image and generating a person image area (bounding box) in which the person is captured. This image recognition unit 211 can be realized, for example, by YOLO (registered trademark), which is an object detection algorithm.

[0019] The tracking unit 212 executes a process of tracking the movement of a person by sequentially identifying the location using the feature amount of the person image area included in each frame image of the video. The code recognition unit 213 identifies the code image included in the frame image and decodes the code image to execute a process of acquiring the color code.

[0020] User management data for managing each user is recorded in the user information storage unit 22. This user management data is recorded when each user is registered. This user management data includes data related to a color code, user ID, name, and affiliation.

[0021] The color code is an identifier for identifying the code image. The user ID is an identifier for identifying the user who carries the code image. The name and affiliation are information about the user's name and affiliation.

[0022] Area management data for managing the area photographed by camera C1 is recorded in area information storage unit 23. This area management data is recorded when the installation location of camera C1 is registered. This area management data records data related to the camera ID and the photographing area.

[0023] The camera ID is an identifier for identifying each camera C1. The image capture area is composed of coordinates that represent the area included in the image captured by this camera in the management area. This image capture area makes it possible to identify the coordinates where the subject (person) is located in the management area.

[0024] Location management data for managing the location of each user is recorded in the location information storage unit 24. This location management data is recorded when the user's location is identified. This location management data includes data on the feature amount, person image ID, user ID, detection time, and location.

[0025] The feature amount is information (numerical values, vectors, etc.) that represents the features of the shape of the subject included in the detected person image area. The person image ID is an identifier that identifies the person image area.

[0026] The user ID is an identifier of the user that is identified by recognizing the code image included in the person image area. The detection time is the date and time when the person image area was detected. The location is information for identifying the location where the person image area is detected, and is expressed, for example, by coordinates or an area identifier.

[0027] (Person tracking processing) Next, the person tracking process will be described with reference to FIG.

[0028] Here, the control unit 21 of the management server 20 executes a frame image acquisition process (step S11). Specifically, the image recognition unit 211 of the control unit 21 acquires a predetermined frame image (captured image) from a moving image captured by the camera C1.

[0029] Next, the control unit 21 of the management server 20 executes person recognition processing (step S12). Specifically, the image recognition unit 211 of the control unit 21 searches for a person in the frame image by object recognition processing.

[0030] Next, the control unit 21 of the management server 20 executes a determination process as to whether a person is included (step S13). Specifically, when the image recognition unit 211 of the control unit 21 detects a person in the frame image, it determines that a person is included.

[0031] If it is determined that no person is included ("NO" in step S13), the control unit 21 of the management server 20 returns to the frame image acquisition process (step S11). Here, the image recognition unit 211 of the control unit 21 acquires the subsequent frame image in the video.

[0032] On the other hand, if it is determined that a person is included ("YES" in step S13), the control unit 21 of the management server 20 executes a process of identifying a person image area (step S14). Specifically, the image recognition unit 211 of the control unit 21 identifies a person image area in which a person is recognized in the frame image.

[0033] 5, a plurality of people P1 and P2 are included in the frame image 500. In this case, person image areas IP1 and IP2 are specified corresponding to the people P1 and P2, respectively.

[0034] Next, the control unit 21 of the management server 20 executes a determination process to determine whether the feature amounts of the person image areas have already been registered (step S15). Specifically, the tracking unit 212 of the control unit 21 calculates the feature amounts of each person image area. Next, the tracking unit 212 checks whether feature amounts highly similar to the calculated feature amounts are recorded in the location information storage unit 24. Here, for example, if the feature amounts match by 80% or more, it is determined that the similarity is high.

[0035] If it is determined that no feature with a high degree of similarity has been recorded and that the feature of the person image area has not already been registered ("NO" in step S15), the control unit 21 of the management server 20 executes a process of assigning a new registration code (step S16). Specifically, the tracking unit 212 of the control unit 21 assigns a person image ID to the feature of the person image area. Next, the tracking unit 212 records location management data including the feature and the person image ID in the location information storage unit 24.

[0036] On the other hand, if a feature with a high degree of similarity is recorded and it is determined that the feature of the person image area is already registered ("YES" in step S15), the control unit 21 of the management server 20 executes a process of identifying the already registered code (step S17). Specifically, the tracking unit 212 of the control unit 21 identifies the person image ID recorded in the location management data in which the feature with a high degree of similarity is recorded in the location information storage unit 24.

[0037] Next, the control unit 21 of the management server 20 executes a process of recording the location of the person (step S18). Specifically, the tracking unit 212 of the control unit 21 identifies the location according to the position of the person image area using the area information storage unit 23. Then, the tracking unit 212 records the location together with the detection time in the location management data. Next, the control unit 21 of the management server 20 returns to the frame image acquisition process (step S11).

[0038] (Code recognition processing) Next, the code recognition process will be described with reference to Fig. 4. This process is performed when a frame image is acquired.

[0039] Here, the control unit 21 of the management server 20 executes a search process for a code image (step S21). Specifically, the code recognition unit 213 of the control unit 21 searches for the code image in the frame image by pattern recognition of the layout pattern of the code image.

[0040] Next, the control unit 21 of the management server 20 executes a determination process as to whether a code image is included (step S22). Specifically, the code recognition unit 213 of the control unit 21 determines that a code image is included when a layout pattern of a code image is detected in the frame image. Note that it is also possible to identify multiple code images in a frame image.

[0041] If it is determined that a code image is not included ("NO" in step S22), the control unit 21 of the management server 20 ends the code recognition process. Then, when a subsequent frame image is acquired, the control unit 21 executes the code recognition process again.

[0042] On the other hand, if it is determined that a code image is included ("YES" in step S22), the control unit 21 of the management server 20 executes a code identification process (step S23). Specifically, the code recognition unit 213 of the control unit 21 decodes the code image to identify the color code.

[0043] Next, the control unit 21 of the management server 20 executes a determination process to determine whether the code image corresponds to a person (step S24). Specifically, the code recognition unit 213 of the control unit 21 compares the detected position of the code image with the arrangement of the person image area. If the detected position of the code image is included in the person image area, it is determined that the code image corresponds to a person.

[0044] If it is determined that the detected position of the code image is outside the person image area and does not correspond to a person ("NO" in step S24), the control unit 21 of the management server 20 ends the code recognition process.

[0045] On the other hand, if it is determined that the detected position of the code image is within the person image area and corresponds to a person ("YES" in step S24), the control unit 21 of the management server 20 executes a user ID identification process (step S25). Specifically, the code recognition unit 213 of the control unit 21 uses the user information storage unit 22 to identify the user ID corresponding to the color code.

[0046] 6, code images CC1 and CC2 are detected in frame image 501. In this case, the code image CC1 in the person image area IP1 of person P1 and the code image CC2 in the person image area IP2 of person P2 are associated with each other, while the code image CC3 outside the person image area is not used to identify the user ID because it is outside the person image area.

[0047] Next, the control unit 21 of the management server 20 executes a process of recording a user ID for the person image (step S26). Specifically, the code recognition unit 213 of the control unit 21 records the user ID in the location management data in which the person image ID of the person image area is recorded.

[0048] (Location confirmation process) When an administrator checks the location of a user, the administrator accesses the management server 20 using the management terminal 10. In this case, the control unit 21 of the management server 20 arranges, on a map of the area to be managed, a movement trajectory that chronologically connects the location of each user recorded in the location information storage unit 24. Then, the control unit 21 outputs the map on which the movement trajectories are arranged to the management terminal 10.

[0049] (Action of this embodiment) Since object recognition and code recognition are used in combination, a person identified by object recognition processing can be tracked using one captured image, and the person can be identified using the code image.

[0050] (Effects of this embodiment) (1) In this embodiment, the control unit 21 of the management server 20 executes a frame image acquisition process (step S11) and a person recognition process (step S12), thereby making it possible to confirm the presence of a person in the captured image.

[0051] (2) In this embodiment, if it is determined that a person is included (if "YES" in step S13), the control unit 21 of the management server 20 executes a process for identifying a person image area (step S14). This allows the area in the frame image in which the person is shown to be identified.

[0052] (3) In this embodiment, if it is determined that the feature amount of the person image area has not been registered (if "NO" in step S15), the control unit 21 of the management server 20 executes a process of assigning a new registration code (step S16). This allows a person image ID to be assigned to a newly photographed person.

[0053] (4) In this embodiment, if it is determined that the feature of the person image area is already registered (YES in step S15), the control unit 21 of the management server 20 executes a process to identify the already registered code (step S17). This makes it possible to track person images with common feature.

[0054] (5) In this embodiment, the control unit 21 of the management server 20 executes a process of recording the location of the person (step S18), thereby making it possible to identify the location of the person using the captured image.

[0055] (6) In this embodiment, the control unit 21 of the management server 20 executes a search process for a code image (step S21), thereby making it possible to identify the code image in the captured image.

[0056] (7) In this embodiment, the control unit 21 of the management server 20 executes a code identification process (step S23) and a determination process (step S24) as to whether the code corresponds to a person. This allows the person recognized by the object recognition process to be associated with the code image.

[0057] (8) In this embodiment, if it is determined that the image corresponds to a person ("YES" in step S24), the control unit 21 of the management server 20 executes a user ID identification process (step S25). This allows the user's location to be managed. Typically, workers in factories wear the same work clothes, making it difficult to identify individuals. It is also possible to track individuals by assigning them unique codes, but this requires additional equipment. For example, tracking using color codes requires installing cameras at a distance where the color codes can be recognized, which requires many cameras. Furthermore, using wireless technology such as RFID requires installing many antennas. In the present invention, a code image is associated with a person and the person is tracked, allowing location management to be performed with relatively simple equipment.

[0058] This embodiment can be modified as follows: This embodiment and the following modifications can be combined and implemented within the scope of technical compatibility. In the above embodiment, the image recognition unit 211 is realized by YOLO, which is an object detection algorithm. The person tracking process is not limited to using YOLO. In the above embodiment, a chameleon code is used as the code image. However, the code image is not limited to a chameleon code as long as it can be recognized by a camera.

[0059] In the above embodiment, a camera is provided for each area. Alternatively, multiple cameras may be placed at different positions in one area. In this case, one person is photographed by multiple cameras.

[0060] As shown in FIG. 7, cameras C11 and C12 that capture the same range are installed at different positions. Then, the control unit 21 of the management server 20 executes a process of acquiring person features using each camera (step S31). Specifically, the image recognition unit 211 of the control unit 21 captures images of people P3 and P4 using each of the cameras C11 and C12. In this case, the camera C11 identifies person image areas IP31 and IP41 of people P3 and P4. Furthermore, the camera C12 identifies person image areas IP32 and IP42 of people P3 and P4. Then, the control unit 21 calculates each image feature from the person image areas IP31, IP41, IP32, and IP42. In this case, the image feature and user ID are associated with each of people P3 and P4.

[0061] Next, the control unit 21 of the management server 20 executes a process of identifying highly similar person feature amounts (step S32). Specifically, the tracking unit 212 of the control unit 21 identifies the location management data 241, 242 having similar feature amounts of the person image areas captured by each camera C11, C12. Then, the location management data 241, 242 are used to track the person image areas according to the similarity of the respective image feature amounts.

[0062] This allows tracking to continue in a complementary manner, since person feature amounts captured by multiple cameras are used. For example, even if the feature amounts of a person image area in an image captured by one camera become dissimilar and tracking becomes impossible, tracking can be continued using the image captured by the other camera. In this case, if one camera detects a person image area with similar person feature amounts and tracking is restored using the captured image, tracking will become possible again using multiple cameras. [Explanation of symbols]

[0063] A1...location management system, C1...camera, 10...management terminal, 20...management server, 21...control unit, 211...image recognition unit, 212...tracking unit, 213...court recognition unit, 22...user information storage unit, 23...area information storage unit, 24...location information storage unit.

Claims

1. A location management system including a camera and a control unit connected to a location information storage unit, The control unit Identifying a person image area in which a person is recognized in the image captured by the camera; calculating a feature amount of a subject included in the person image area, and associating a previously registered person image ID with the person image area if a similarity between the feature amount and a previously registered feature amount recorded in the location information storage unit is equal to or greater than a predetermined threshold, and assigning a new person image ID to the person image area if the similarity is less than the predetermined threshold; a first process of sequentially identifying the location of the person in accordance with the position of the person image area, regardless of whether a code image is detected or not, and recording the identified location in the location information storage unit; a second process for recording a code corresponding to the code image in the location information storage unit in association with the time series of the location corresponding to the person image ID when the code image is detected in the person image area.

2. The location management system according to claim 1 , wherein the control unit connects the location positions in a time series to identify a movement trajectory.

3. The control unit Recognizing the same person using the plurality of cameras and specifying the person image areas, 3. The location management system according to claim 1, wherein when the code image is detected in the person image area of ​​the image captured by at least one of the cameras, a code corresponding to the code image is associated with each of the person image areas.

4. A method for managing a location using a location management system including a camera and a control unit connected to a location information storage unit, comprising: The control unit Identifying a person image area in which a person is recognized in the image captured by the camera; calculating a feature amount of a subject included in the person image area, and associating a previously registered person image ID with the person image area if a similarity between the feature amount and a previously registered feature amount recorded in the location information storage unit is equal to or greater than a predetermined threshold, and assigning a new person image ID to the person image area if the similarity is less than the predetermined threshold; a first process of sequentially identifying the location of the person in accordance with the position of the person image area, regardless of whether a code image is detected or not, and recording the identified location in the location information storage unit; a second process for recording a code corresponding to the code image in the location information storage unit in association with the time series of the location corresponding to the person image ID when the code image is detected in the person image area.

5. A program for managing a location using a location management system including a camera and a control unit connected to a location information storage unit, The control unit Identifying a person image area in which a person is recognized in the image captured by the camera; calculating a feature amount of a subject included in the person image area, and associating a previously registered person image ID with the person image area if a similarity between the feature amount and a previously registered feature amount recorded in the location information storage unit is equal to or greater than a predetermined threshold, and assigning a new person image ID to the person image area if the similarity is less than the predetermined threshold; a first process of sequentially identifying the location of the person in accordance with the position of the person image area, regardless of whether a code image is detected or not, and recording the identified location in the location information storage unit; A location management program characterized by functioning as a means for executing a second process in which, when the code image is detected in the person image area, a code corresponding to the code image is associated with the time series of the location corresponding to the person image ID and recorded in the location information storage unit.

Images