Surveillance system

Abstract

To provide a monitoring system with which it is possible to conduct monitoring by an image in which consideration is taken for the privacy of an unvaccinated patient.SOLUTION: A monitoring system according to the present example comprises: an imaging device 10 for imaging a prescribed monitoring area; an image processing device 20 for determining, for each person included in the image captured by the imaging device 10, the necessity for anonymizing the person, and performing an anonymization process of processing an image portion of the person for whom anonymization is determined to be necessary; and a digital signage display device 30 for passengers and a display device 40 for facility operators on which the image having been processed by the image processing device 20 is displayed. The image processing device 20 refers to person data that includes the vaccination state of preregistered persons to confirm the vaccination state of the person included in the image, and identifies a person not confirmed to have been vaccinated (i.e., an unvaccinated parson or a person with unknown vaccination status), as the person who needs to be anonymized.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a surveillance system that displays an image of a predetermined surveillance area captured by an imaging device on a display device. [Background technology]

[0002] In recent years, human detection and object recognition technologies using deep learning etc. have been developed. These technologies make it possible to automatically recognize a person's physical characteristics such as gender and age, whether they are wearing a mask, and whether they are using care products such as a wheelchair or a white cane.

[0003] To utilize such human detection and object recognition technologies, surveillance cameras are being installed to ensure safety and security in facility operations. On the other hand, consideration must be given to the privacy of users in vaccination sites for infectious diseases that have spread in recent years, as well as in hospitals, commercial facilities, changing rooms and restrooms in airport facilities, or in their vicinity. For this reason, there are places where surveillance cameras cannot be installed, even for the purpose of ensuring safety and security in facility operations or improving customer satisfaction. Therefore, in the past, anonymized videos that take privacy into consideration were created after object recognition and human behavior analysis, and used for video surveillance.

[0004] The prior art in the technical field of the present invention includes the following: For example, Patent Document 1 discloses an invention that protects personal privacy by reducing or modifying personal privacy information while maintaining a state in which the identity of a person can be determined between multiple cameras. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2006-287287 A Summary of the Invention [Problem to be solved by the invention]

[0006] There were concerns that surveillance footage that overly anonymized characteristics such as vaccination status and whether or not a mask was worn, as well as personal and physical characteristics, would make it difficult to properly monitor the three Cs (closed spaces, crowded places, and close contact) as a measure against infectious diseases. In addition, the vaccination status of facility personnel is synonymous with indicating the cleanliness of the facility itself and whether it is a safe and secure environment, so for facility operators, disclosing the vaccination status of facility employees had the advantage of improving customer satisfaction. However, displaying footage that does not take into consideration the wishes of facility employees not to get vaccinated was problematic from the perspective of protecting the personal information of unvaccinated people.

[0007] The present invention has been made in consideration of the above-mentioned conventional circumstances, and has an object to provide a surveillance system that is capable of monitoring using images that respects the privacy of unvaccinated persons. [Means for solving the problem]

[0008] In order to achieve the above object, a surveillance system according to one aspect of the present invention is configured as follows: That is, the surveillance system according to the present invention includes an imaging device that captures an image of a predetermined surveillance area, an image processing device that determines the necessity of anonymization of each person included in an image captured by the imaging device and performs an anonymization process to process image portions of the person determined to require anonymization, and a display device that displays the image after processing by the image processing device, and is characterized in that the image processing device checks the vaccination status of the person included in the image by referring to person data including the vaccination status of the person registered in advance, and determines that a person who cannot be confirmed to have been vaccinated is a person who requires anonymization.

[0009] Here, in the surveillance system according to the present invention, the person data is associated with facial image of a pre-registered person or facial feature data including its features, and the image processing device can be configured to identify a person included in an image captured by the imaging device through facial recognition processing using the facial feature data, and to determine the need for anonymization based on the person data for the identified person.

[0010] In addition, in the surveillance system of the present invention, the personal data is associated with worn item images or worn item feature data including the features of the worn items corresponding to the affiliation of the person registered in advance, and the image processing device can be configured to identify the person included in an image captured by the imaging device by face recognition processing using the worn item feature data, when the image processing device is able to identify the affiliation of the person, by face recognition processing using the face data, and to determine the need for anonymization based on the personal data for the identified person.

[0011] In addition, in the surveillance system according to the present invention, the image processing device can be configured to include, among people included in an image captured by the imaging device, people who are not the subject of personal data, as people who need to be anonymized.

[0012] Furthermore, the surveillance system according to the present invention may be configured to cause the display device to display an alarm when the condition that the number of people who need to be anonymized within a predetermined range is equal to or greater than a predetermined value is satisfied.

[0013] Furthermore, the monitoring system according to the present invention may be configured to cause the display device to display an alarm when the state satisfying the above condition continues for a predetermined period of time or longer. Effect of the Invention

[0014] According to the present invention, it is possible to provide a surveillance system capable of monitoring unvaccinated persons using images while respecting their privacy. [Brief description of the drawings]

[0015] [Figure 1] 1 is a diagram illustrating an example of the configuration of a monitoring system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating an example of a flowchart of a process performed by the monitoring system of FIG. [Diagram 3] FIG. 2 is a diagram showing a display example of a passenger electronic signage display device. [Figure 4] FIG. 13 is a diagram showing a display example of a facility operator display device. [Diagram 5] FIG. 13 is a diagram showing a display example of a facility operator display device. [Figure 6] FIG. 13 is a diagram showing a display example of a facility operator display device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] An embodiment of the present invention will be described with reference to the drawings. Fig. 1 shows an example of the configuration of a surveillance system according to an embodiment of the present invention. The surveillance system in Fig. 1 includes an imaging device 10, an image processing device 20, a passenger electronic signage display device 30, a facility operator display device 40, and an image recording device 50. These devices are each connected to a network N. The surveillance system of this example is intended for application to facilities such as stations and airports, but can be applied to various other facilities.

[0017] The imaging device 10 has a camera configuration and is arranged in various places. For example, it is arranged as a surveillance camera so as to capture a predetermined surveillance area. The imaging device 10 can be configured as a camera that obtains information by forming an image of incident light on an imaging element via a lens and an aperture. Examples of the imaging element here include a CCD (Charge-Coupled Device) image sensor and a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The imaging device 10 captures images at, for example, 3 frames per second (3 fps) or more, and the information is sent to an image processing device 20 or an image recording device 50. A plurality of imaging devices 10 can be installed depending on the situation such as the shape and size of the surveillance area.

[0018] The image processing device 20 includes a control unit 21 , a user interface unit 22 , an image analysis unit 23 , an anonymous image generation unit 24 , and an alarm generation unit 25 .

[0019] The control unit 21 can exchange information with other devices through the network N, and controls the user interface unit 22, the image analysis unit 23, the anonymized image generation unit 24, and the alarm generation unit 25. When a user inputs or operates the passenger electronic signage display device 30, the facility operator display device 40, etc., the user interface unit 22 performs processing such as accepting the input or operation information. The image analysis unit 23 analyzes the image captured by the imaging device 10, and performs processing to determine the necessity of anonymization for each person according to the vaccination status of each person included in the image. The anonymized video generation unit 24 performs processing to apply processing to the image portion of the image captured by the imaging device 10 for the person determined to require anonymization. The alarm generation unit 25 performs processing to generate an alarm according to the congestion status of unvaccinated people.

[0020] The passenger electronic signage display device 30 and the facility operator display device 40 are display devices capable of displaying the contents processed by the image processing device 10. The passenger electronic signage display device 30 is a display device provided to users of facilities such as stations and airports, and is installed at any location that the facility users can visit. The facility operator display device 40 is a display device provided to those involved in the facility (operators and employees), and is installed at a location that can be used only by those involved in the facility, such as a monitoring room. These display devices 30, 40 perform display using, for example, a liquid crystal display (LCD), an organic electroluminescence (OEL) display, a touch panel, or the like. The display devices 30, 40 may also be equipped with operation buttons. The display devices 30, 40 can be installed in any number as required.

[0021] The image recording device 50 is a device that records images captured by the imaging device 10. The image recording device 50 may also record images or results after processing by the image processing device 10. As the image recording device 50, for example, a hard disk drive (HDD), a solid state drive (SSD), a digital data storage (DDS), or other suitable method can be applied as necessary.

[0022] The network N is a line capable of data communication connecting each device. The network N can be applied regardless of the type of line, such as a dedicated line, an intranet, or an IP network such as the Internet.

[0023] FIG. 2 shows an example of a flowchart of the process by the monitoring system of this embodiment. Here, it is assumed that person data in which facility personnel are registered in advance is stored in a database (not shown) accessible from the image processing device 20. In addition to the person's name, this person data includes data indicating vaccination status. In addition, various feature data representing the characteristics of the person are associated with the person data. Examples of the feature data include physical feature data indicating the person's age, sex, whether or not they wear glasses, whether or not they wear hearing aids, whether or not they can walk, etc., facial feature data required for face recognition processing to identify the person, and wearable item feature data required for wearable item recognition processing to identify the department (category) to which the person belongs. The facial feature data includes, for example, a person's face image or its feature amount. The wearable item feature data includes, for example, images or its feature amount of hats, uniforms, badges, armbands, belongings, etc. according to the department to which the person belongs.

[0024] The image processing device 20 analyzes the images captured by the imaging device 10 using the image analysis unit 23. Specifically, the image analysis unit 23 first detects a person included in each frame of an image captured by the imaging device 10 (step S11). Next, the image analysis unit 23 extracts features related to each person detected from the image captured by the imaging device 10 (step S12). In this example, the features extracted are facial features and features of items worn by the person.

[0025] Next, the image analysis unit 23 performs a wear item recognition process to match the feature amount of the wear of the person detected from the image with the wear item feature data, and determines whether the person is a facility-related person who belongs to the facility (step S13). For example, if wear item feature data is found that matches the feature amount of the wear items extracted from the image to a predetermined value or more, the person is determined to be a facility-related person. On the other hand, if no wear item feature data that meets the above requirements is found, the person is determined to be not a facility-related person (i.e., a facility user).

[0026] Next, the image analysis unit 23 performs face recognition processing on the person determined to be a facility-related person to match the facial feature amount extracted from the image with the facial feature data to identify the person, and determines the vaccination status based on the person data associated with the corresponding facial feature data (step S14). For example, when facial feature data is found that matches the facial feature amount extracted from the image to a predetermined value or more, the image analysis unit 23 determines whether the person has been vaccinated or not by referring to the person data associated with the facial feature amount. On the other hand, when no facial feature data that meets the above requirements is found, the image analysis unit 23 determines that the vaccination status is unknown. Furthermore, the image analysis unit 23 determines that the vaccination status is unknown for a person determined not to be a facility-related person (i.e., a facility user) without performing face recognition processing.

[0027] Next, the image analysis unit 23 performs a process of recording the judgment results of steps S13 and S14, i.e., the judgment results of whether or not the person is a facility-related person and the judgment results of the vaccination status, in memory (step S15). The above process is repeatedly performed for all people included in the images captured by the imaging device 10 (step S16).

[0028] Next, the anonymized image generating unit 24 processes the image of a person who has not been confirmed as vaccinated by the analysis in the image analyzing unit 23, that is, a person who has not been vaccinated or whose vaccination status is unknown, as a person who needs to be anonymized, and performs a process of anonymizing the image of the person by processing the image portion of the person (step S17). For example, in anonymizing a face, the face image or face part image of the target person is compared with a plurality of types of abstract face images or abstract face part images registered in advance (for example, feature comparison), and replaced and synthesized with the abstract face image or abstract face part image with the highest similarity. In the same manner, the body and clothes other than the face may be anonymized. Alternatively, one or more avatar images may be prepared in association with age and gender, and replaced and synthesized with the avatar image according to the age and gender of the target person.

[0029] Next, the alarm generation unit 25 determines whether an alarm is necessary depending on the degree of congestion of people who need to be anonymized (i.e., people who have not been vaccinated or whose vaccination status is unknown), and issues an alarm if it is determined that an alarm is necessary (step S18). For example, it determines that an alarm is necessary when the condition that the number of people who need to be anonymized within a predetermined range is equal to or greater than a predetermined value is satisfied. Alternatively, it determines that an alarm is necessary when the state that satisfies the above condition continues for a predetermined period of time or more. In addition, the behavior of people who need to be anonymized may be analyzed by detecting skeletal structure, etc., and used for the above determination.

[0030] The anonymized image generated by the anonymized image generating unit 24 is transmitted to the passenger electronic signage display device 30 and the facility operator display device 40 and displayed on their screens. When an alarm is generated by the alarm generating unit 25, the data is transmitted to the passenger electronic signage display device 30 and the facility operator display device 40 and displayed on their screens.

[0031] 3 shows an example of the display of the passenger electronic signage display device 30. The screen of the passenger electronic signage display device 30 in this example has a display section 31 showing the vaccination status at required locations in the facility, a display section 32 showing a map of the facility, and a display section 33 showing an anonymized image obtained by anonymizing an image captured by one of the cameras (imaging device 10) in the facility.

[0032] For example, the display unit 31 displays a pie chart showing the ratio of vaccinated and unvaccinated facility personnel in the vicinity of each of three gates A, B, and C installed in the facility. This ratio can be calculated by analyzing, with the image processing device 10, images captured by an imaging device 10 installed in the vicinity of the area, and identifying vaccinated and unvaccinated facility personnel (including those with unknown vaccination status).

[0033] The cameras to be displayed by the display unit 33 are automatically switched at regular time intervals, for example. Also, the facility user may be able to select the cameras to be displayed by the display unit 33 by operating the passenger electronic signage display device 30. For example, when the facility user touches the map displayed on the display unit 32, anonymized images may be obtained for the cameras installed at the location in the facility that corresponds to the touched position, and the anonymized images may be displayed on the display unit 33.

[0034] 4, 5 and 6 show examples of displays on the facility operator's display device 40. The screen of the facility operator's display device 40 in this example has a display unit 41 showing a map of the facility and the vaccination status at required locations in the facility, a display unit 42 showing an image captured by any camera (imaging device 10) in the facility or anonymized image, an operation unit 43 that accepts an operation from facility personnel to select a camera to be displayed by the display unit 42, a display unit 44 showing the occurrence status of an alarm, and an operation unit 45 that accepts an operation from facility personnel to check the details of an alarm.

[0035] As shown in Fig. 4, the facility operator's display device 40 normally uses an area occupying most of the screen as a display section 41 for a map of the facility and the vaccination status at required locations in the facility, and when a camera selection operation is received at the operation section 43, it switches to a display section 42 for an image captured by that camera (imaging device 10) or an anonymized image, as shown in Fig. 5 or 6. Fig. 5 is an example of a format in which an image captured by the camera is displayed as is on the display section 42, and Fig. 6 is an example of a format in which an image captured by the camera is anonymized and then displayed on the display section 42. Which display format is adopted may be determined in advance, or may be dynamically switched depending on the authority of the user of the facility operator's display device 40, etc.

[0036] The occurrence of alarms in the facility is displayed in chronological order on the display unit 44. When the operation unit 45 is operated with any alarm selected on the display unit 44, details of the selected alarm are further displayed. At this time, an image or an anonymous image at the time of the alarm may be read from the image recording device 50 and displayed on the display unit 42.

[0037] As described above, the surveillance system of this example comprises an imaging device 10 that captures an image of a specified surveillance area, an image processing device 20 that determines the need for anonymization of each person included in the image captured by the imaging device 10 and performs an anonymization process to process the image portions of the people determined to require anonymization, and a passenger electronic signage display device 30 and a facility operator display device 40 that display the images after processing by the image processing device 20. The image processing device 20 refers to person data including the vaccination status of pre-registered people, confirms the vaccination status of people included in the image, and determines that people who cannot be confirmed to have been vaccinated (i.e. people who have not been vaccinated or whose vaccination status is unknown) are people who require anonymization.

[0038] Therefore, according to the monitoring system of this example, the passenger electronic signage display device 30 can present the vaccination status to facility users, so that if the vaccination status of facility personnel is low, the possibility of virus infection from facility personnel is low, and it is possible to inform facility users that the facility is safe and secure. On the other hand, for unvaccinated or unknown persons, their vaccination status is anonymized to respect the individual's privacy and intention of not being vaccinated, making it possible to monitor using images that take into consideration the privacy of unvaccinated persons.

[0039] Furthermore, in the monitoring system of this example, when the image processing device 20 is able to identify the affiliation (i.e., that the person is a facility-related person) of a person included in an image captured by the imaging device 10 through a wearing item recognition process using the wearing item feature data, the image processing device 20 is configured to identify the person through a face recognition process using the face data, and to determine the need for anonymization based on the person data of the identified person. With this configuration, it is possible to omit face recognition processing for people who are found not to be facility-related people through the wearing item recognition process, thereby making it possible to efficiently determine the vaccination status.

[0040] In this example, anonymization processing is performed to process the image portion of an unvaccinated person, but the image portion of a vaccinated person may also be processed. That is, for example, facial images of facility personnel without infectious disease control products such as masks or face guards are captured and stored, and the facial images of vaccinated persons are replaced with facial images without infectious disease control products. This makes it easier for facility personnel to identify people in the images when performing surveillance work.

[0041] In this example, all facility users are anonymized as people with unknown vaccination status, but facility users who use wheelchairs, white canes, and other nursing care items may be excluded from the anonymization process for the purpose of providing support to the users. This makes it easier to recognize people who need support, so facility personnel can quickly go to those people.

[0042] Although the embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and various modifications such as omissions and substitutions can be made without departing from the gist of the present invention. These embodiments and modifications are included in the scope and gist of the invention described in this specification, etc., and are included in the scope of the invention described in the claims and their equivalents.

[0043] Furthermore, the present invention can be provided not only as devices such as those described above or as a system composed of these devices, but also as methods executed by these devices, programs for causing a processor to realize the functions of these devices, and storage media for storing such programs in a computer-readable format. [Industrial Applicability]

[0044] The present invention can be used in a surveillance system that displays an image of a predetermined surveillance area captured by an imaging device on a display device. [Explanation of symbols]

[0045] 10: Imaging device, 20: Image processing device, 21: Control unit, 22: User interface unit, 23: Image analysis unit, 24: Anonymous image generation unit, 25: Alarm generation unit, 30: Passenger electronic sign display device, 40: Facility operator display device, 50: Image recording device

Claims

1. An imaging device that captures an image of a predetermined monitoring area; an image processing device that performs an anonymization process by referring to personal data including the vaccination status of pre-registered persons, confirming the vaccination status of each person included in the image captured by the imaging device, determining that a person who has not been confirmed to have been vaccinated is a person who needs to be anonymized, and processing the image portion of that person; a display device for displaying an image after processing by the image processing device, The person data is associated with face feature data including a face image or a feature amount of the pre-registered person, and wearing item feature data including a wearing item image or a feature amount of the wearing item according to an affiliation of the pre-registered person, The surveillance system is characterized in that, when the image processing device can identify the affiliation of a person included in an image captured by the imaging device through wearing item recognition processing using the wearing item feature data, the image processing device identifies the person through face recognition processing using the facial feature data, and determines the need for anonymization based on the personal data of the identified person.

2. 2. The monitoring system according to claim 1, The surveillance system is characterized in that the image processing device includes, among people included in an image captured by the imaging device, people who are not the subject of the personal data, as people who need to be anonymized.

3. 3. The monitoring system according to claim 1, A monitoring system characterized in that, when a condition that the number of people who need to be anonymized within a predetermined range is equal to or greater than a predetermined value is satisfied, an alarm is displayed on the display device.

4. 4. The monitoring system according to claim 3, a monitoring system that displays an alarm on the display device when the state satisfying the condition continues for a predetermined period of time or longer.

Images