System, server device, server device control method, and program

Abstract

To provide a system for appropriately protecting privacy of an imaged person.SOLUTION: A system includes: a camera device for imaging a predetermined area; a terminal possessed by a user; and a server device for storing image data captured by the camera device and acquisition date and time of the image data in association with each other. When becoming communicable with the camera device, the terminal transmits, to the camera device, a user-information notification including user information on the user. In response to reception of the user information notification, the camera device transmits, to the server device, the user information and first image data obtained by imaging the predetermined area. When the first image data includes a user region corresponding to the user, the server device performs, on the user region, privacy protection processing for protecting privacy of the user to generate second image data. The server device stores the generated second image data.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a system, a server device, a control method for the server device, and a storage medium.

Background Art

[0002] Surveillance cameras are installed in various places for purposes such as disaster prevention. In recent years, technological development using images (videos) obtained by surveillance cameras and the like has been promoted.

[0003] For example, Patent Document 1 describes identifying a person corresponding to a traffic line. The traffic line management device of Patent Document 1 includes a traffic line management unit, a time zone specifying unit, a reception history acquisition unit, and a person identification unit. The traffic line management unit identifies a traffic line indicating the position of a person at each time in an action area where one or more target persons act. The time zone specifying unit specifies a time zone in which a person stays in a predetermined area included in the action area based on the identified traffic line. The reception history acquisition unit acquires reception history information including a terminal ID for identifying a terminal, a beacon ID received from a transmitter installed at a position corresponding to a predetermined area by each of a plurality of terminals held by a plurality of target persons, and the time when the beacon ID was received. The person identification unit identifies, among a plurality of target persons, a target person highly likely to be staying in a predetermined area in the specified time zone as a target person corresponding to the identified traffic line based on the reception history information and information associating the terminal ID with the target person.

[0004] Patent Document 2 describes that a user can simultaneously grasp the activity status of a person over a predetermined period and the activity status of the person at a certain moment using a single monitoring screen. The system of Patent Document 1 includes a position information acquisition unit, a statistical information acquisition unit, a heat map image generation unit, a mask image generation unit, and a video output control unit. The position information acquisition unit detects a person from the video of the monitoring area and acquires the position information of the person. The statistical information acquisition unit performs temporal statistical processing on the position information of the person and acquires statistical information regarding the staying status of the person according to the setting of the target period of the statistical processing. The heat map image generation unit generates a heat map image in which the statistical information is visualized. The mask image generation unit generates a mask image corresponding to the image area of the person at each predetermined time based on the position information of the person. The video output control unit generates and outputs a monitoring video in which the heat map image and the mask image are superimposed on a background image at each predetermined time.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] The entity (individual, group, etc.) that installs the surveillance camera uses the images obtained from the surveillance camera for disaster prevention and crime prevention. The surveillance camera photographs a predetermined area and acquires image data (video data). Here, the users photographed by the surveillance camera may feel that their privacy is not protected against an organization or the like that collects image data (video data) including an unspecified number of individuals.

[0007] The main object of the present invention is to provide a system, a server device, a control method of the server device, and a storage medium that contribute to appropriately protecting the privacy of a person to be photographed.

Means for Solving the Problems

[0008] According to a first aspect of the present invention, there is provided a system including: a camera device that photographs a predetermined area; a terminal possessed by a user; and a server device that stores, in association with each other, image data photographed by the camera device and the acquisition date and time of the image data. When the terminal becomes communicable with the camera device, the terminal transmits a user information notification including user information about the user to the camera device. The camera device transmits, to the server device, first image data obtained by photographing the predetermined area in response to the reception of the user information notification and the user information. When the first image data includes a user area corresponding to the user, the server device generates second image data by performing a privacy protection process for protecting the privacy of the user with respect to the user area, and stores the generated second image data.

[0009] According to a second aspect of the present invention, there is provided a server device including: a storage unit that stores, in association with each other, image data photographed by a camera device that is capable of photographing a predetermined area and becomes communicable with a terminal possessed by a user, and the acquisition date and time of the image data; a reception unit that receives, from the camera device, first image data obtained by photographing the predetermined area in response to the reception of a user information notification including user information about the user by the camera device and the user information; and an image data control unit that generates second image data by performing a privacy protection process for protecting the privacy of the user with respect to the user area when the first image data includes a user area corresponding to the user, and stores the generated second image data.

[0010] According to a third aspect of the present invention, in a server device, when a camera device capable of photographing a predetermined area and communicating with a terminal possessed by a user receives a user information notification including user information regarding the user from the terminal, the server device stores the image data photographed by the camera device in association with the acquisition date and time of the image data. When the server device receives the first image data obtained by the camera device photographing the predetermined area in response to the reception of the user information notification and the user information from the camera device, and the first image data includes a user area corresponding to the user, the server device generates second image data by performing privacy protection processing for protecting the privacy of the user with respect to the user area, and stores the generated second image data. A control method for the server device is provided.

[0011] According to a fourth aspect of the present invention, a computer-readable storage medium stores a program for causing a computer mounted on a server device to execute a process of storing, in association with each other, image data photographed by a camera device that can photograph a predetermined area and can communicate with a terminal possessed by a user, and the acquisition date and time of the image data when the computer receives a user information notification including user information regarding the user from the terminal; a process of receiving, from the camera device, the first image data obtained by the camera device photographing the predetermined area in response to the reception of the user information notification and the user information; and a process of generating second image data by performing privacy protection processing for protecting the privacy of the user with respect to the user area when the first image data includes a user area corresponding to the user, and storing the generated second image data.

Advantages of the Invention

[0012] According to each aspect of the present invention, there are provided a system, a server device, a control method for the server device, and a storage medium that contribute to appropriately protecting the privacy of a person being photographed. Note that the effects of the present invention are not limited to the above. Instead of or together with the above effects, other effects may be achieved by the present invention.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

MODE FOR CARRYING OUT THE INVENTION

[0014] First, an overview of an embodiment will be described. Note that the reference numerals in the drawings appended to this overview are for convenience of each element as an example to assist understanding, and the description of this overview is not intended to be limiting in any way. Also, unless otherwise specified, the blocks described in each drawing represent a configuration of functional units, not a configuration of hardware units. The connection lines between the blocks in each figure include both bidirectional and unidirectional ones. For a unidirectional arrow, it schematically shows the flow of the main signal (data) and does not exclude bidirectionality. In this specification and the drawings, for elements that can be similarly described, duplicate description may be omitted by assigning the same reference numeral.

[0015] The system according to an embodiment includes a camera device 101 that captures a predetermined area, a terminal 102 held by a user, and a server device 103 that stores the image data captured by the camera device 101 in association with the acquisition date and time of the image data (see FIG. 1).

[0016] FIG. 2 is a flowchart for explaining the operation outline of the system according to an embodiment. When the terminal 102 becomes communicable with the camera device 101, it transmits a user information notification including user information regarding the user to the camera device 101 (step S1). In response to receiving the user information notification, the camera device 101 transmits the first image data obtained by capturing a predetermined area and the user information to the server device 103 (step S2). When the first image data includes a user area corresponding to the user, the server device 103 generates second image data by performing a privacy protection process for protecting the privacy of the user with respect to the user area (step S3). The server device 103 stores the generated second image data (step S4).

[0017] In the above system, the terminal 102 provides the information of the user (user information; for example, gender, age, etc.) to the camera device 101. When the user who has provided the user information is photographed by the camera device 101, privacy protection processing is performed on the user area of the user (the area within the image data generated by imaging the user; the area where the user appears). Specifically, a mosaic is applied or the area is filled in so that the user appearing in the image data cannot be identified (specified). In this way, since the image of the person being photographed is masked (hidden) in response to providing the user information, the privacy of the person being photographed is appropriately protected. In exchange for providing the user information (for example, gender or age), since the image of the user is masked, the anxiety of the user with respect to an organization that collects image data (video data) including an unspecified number of individuals is reduced.

[0018] Specific embodiments will be described in more detail below with reference to the drawings.

[0019] [First Embodiment] The first embodiment will be described in more detail with reference to the drawings.

[0020] [Configuration of the System] FIG. 3 is a diagram showing an example of the schematic configuration of an information processing system (video analysis system) according to the first embodiment. As shown in FIG. 3, the information processing system includes a server device 10 and a plurality of camera devices 20-1 and 20-2.

[0021] In the following description, when there is no particular reason to distinguish between the camera devices 20-1 and 20-2, they are simply referred to as "camera device 20".

[0022] As shown in FIG. 3, the user has a terminal 30. For example, the terminal 30 is a mobile terminal such as a smartphone or a tablet.

[0023] The server device 10 and each camera device 20 are connected by wired or wireless communication means and are configured to be able to communicate with each other. Also, the terminal 30 held by the user is configured to be able to communicate with a server or the like on the network via a mobile line or the like. Further, the terminal 30 is configured to be able to communicate with the camera device 20 by proximity wireless communication means such as Bluetooth (registered trademark).

[0024] The camera device 20 is installed in public areas such as stations and airports, inside buildings of companies, within premises, etc. for the purposes of crime prevention, disaster countermeasures, accident countermeasures, etc. The camera device 20 photographs a predetermined area. For example, the camera device 20-1 photographs the shooting area A1. The camera device 20-2 photographs the shooting area A2.

[0025] The camera device 20 transmits image data to the server device 10 at regular intervals or at a predetermined timing. At that time, the camera device 20 transmits the camera ID to the server device 10 together with the image data. Note that the camera ID is an ID for identifying the camera device 20 included in the information processing system. The MAC (Media Access Control) address or IP (Internet Protocol) address of the camera device 20 can be used as the camera ID.

[0026] The server device 10 is a device that controls the image data received from the camera device 20. The server device 10 stores the image data received from each camera device 20. More specifically, the server device 10 stores the image data photographed by the camera device 20 in association with the acquisition date and time of the image data.

[0027] Furthermore, the server device 10 executes a process for protecting the privacy of the user photographed by the camera device 20 (hereinafter referred to as privacy protection process) as necessary. The details of the privacy protection process will be described later.

[0028] Note that the configuration of the information processing system shown in FIG. 3 is an example and is not intended to limit the configuration. For example, the information processing system may include a plurality of server devices 10. Also, although two camera devices 20 are illustrated in FIG. 3, the number of camera devices 20 is not intended to be limited. The information processing system only needs to include at least one or more camera devices 20.

[0029] [Operation Outline] Subsequently, the operation outline of the information processing system according to the first embodiment will be described.

[0030] [Registration of Attribute Information] Users who are reluctant to have their face or body appear in the images captured by the camera device 20 make prior settings to protect their privacy. Specifically, the user registers attribute information in the terminal 30 using a privacy mask application (hereinafter, may also be simply referred to as an application or app) installed in the terminal 30.

[0031] Specifically, the user operates the terminal 30 to acquire their own biometric information (e.g., face image). For example, the user acquires a face image by taking a so-called self-portrait. Also, the user operates the terminal 30 to photograph an identity document (e.g., driver's license, passport, etc.) on which the biometric information is described.

[0032] The terminal 30 (privacy mask application) performs authentication processing using the user's biometric information and the biometric information described in the identity document. For example, the terminal 30 performs one-to-one authentication using the face image acquired by self-portrait and the face image described in the passport.

[0033] When the authentication is successful, the terminal 30 acquires information indicating the user's attributes (attribute information; person annotation, e.g., gender, age, or a combination thereof, or nationality) from the identity document. For example, the terminal 30 acquires attribute information such as "male in his 40s" or "female in her 30s" from the identity document.

[0034] When the generation of the attribute information is successful, the terminal 30 generates a user ID for identifying the user. The terminal 30 stores the user ID and the attribute information.

[0035] As shown in FIG. 4, each camera device 20 photographs a predetermined photographing area. For example, the camera device 20 photographs the photographing area several times per second. Each time the camera device 20 photographs the photographing area, the camera device 20 transmits the obtained image data to the server device 10. At that time, the camera device 20 transmits an "image data storage request" including its own camera ID and the image data to the server device 10.

[0036] In this way, the camera device 20 transmits the image data to the server device 10 in real time. The server device 10 stores the image data for each camera device 20 (for each camera ID). Note that the camera device 20 may not transmit the image data to the server device 10 in real time, and may transmit two or more pieces of image data to the server device 10 collectively. For example, the camera device 20 may transmit a plurality of pieces of image data to the server device 10 collectively at a predetermined interval (for example, every 10 seconds).

[0037] Here, a user who wants to appropriately protect his or her own privacy goes out with the terminal 30. At that time, the user starts the privacy mask application of the terminal 30.

[0038] When the terminal 30 becomes communicable with the camera device 20, the terminal 30 transmits a "user information notification" including information about the user (the owner of the terminal 30) to the camera device 20. More specifically, when the terminal 30 becomes communicable with the camera device 20 by short-range wireless communication such as Bluetooth (registered trademark), the terminal 30 transmits the "user information notification" to the camera device 20 at regular intervals or at a predetermined timing (see FIG. 5).

[0039] The user information notification includes the user ID, attribute information, date and time information (current time), and location information of the terminal 30 (terminal location) of the user who holds the terminal 30. In the following description, the information included in the user information notification (user ID, attribute information, date and time information, and terminal location) is referred to as "user information".

[0040] When receiving the user information notification, the camera device 20 acquires image data by photographing the shooting area. The camera device 20 transmits the image data obtained by shooting to the server device 10. More specifically, the camera device 20 transmits an image data storage request including the camera ID, the image data, and the user information received from the terminal 30 to the server device 10.

[0041] In this way, the camera device 20 transmits the first image data obtained by photographing a predetermined area in response to the reception of the user information notification from the terminal 30 and the user information acquired from the terminal 30 to the server device 10.

[0042] When the user corresponding to the user information notification is included in the image data (when the user is photographed by the camera device 20), the server device 10 identifies the position of the user in the image (user position). The server device 10 identifies the position of the user (position in the image data) based on the terminal location included in the user information notification transmitted from the terminal 30.

[0043] In the following description, the coordinate system represented by latitude and longitude (latitude and longitude coordinate system) is referred to as the "actual coordinate system". Also, the coordinate system in the image data (coordinate system with the center or lower left of the image data as the origin) is referred to as the "image coordinate system". The terminal 30 transmits the terminal location (X, Y) in the actual coordinate system to the camera device 20. The server device 10 identifies the user position (P, Q) of the user subject to privacy protection using the image coordinates. That is, the terminal 30 transmits the position information (coordinate information) of the user in the actual coordinate system to the camera device 20, and the server device 10 manages the position of the user subject to privacy protection with the position information (coordinate information) in the image coordinate system.

[0044] For example, consider a case where person 41 and person 42 shown in FIG. 6 move within the shooting area of the camera device 20. Person 41 moves passing through terminal positions (X11, Y11), (X12, Y12), and (X13, Y13) between time t1 and time t3. Similarly, person 42 moves passing through terminal positions (X21, Y21), (X22, Y22), and (X23, Y23) between time t1 and time t3.

[0045] Here, person 41 is a user who possesses terminal 30. Terminal 30 transmits a user information notification including the above three terminal positions to the camera device 20 at each of times t1, t2, and t3. That is, person 41 is a user who desires to protect their privacy, and the terminal 30 possessed by the person 41 transmits a user information notification to the camera device 20. On the other hand, person 42 does not possess terminal 30. Therefore, a user information notification regarding person 42 is not transmitted to the camera device 20.

[0046] The server device 10 identifies the user position (position within the image data) of person 41, who is the possessor of the terminal 30 that transmits the user information notification. In the example of FIG. 6, the server device 10 identifies the position of person 41 in the image coordinate system at each of times t1 to t3.

[0047] When the user's position is identified, the server device 10 executes a privacy protection process to protect the privacy of the user. Specifically, the server device 10 makes the user unidentifiable by performing image processing on the area of the user included in the image data (hereinafter referred to as the user area). For example, the server device 10 masks (hides) the user area by applying a mosaic to the user area or filling the user area.

[0048] Note that the user area is an area composed of pixels corresponding to the user's face, body, etc. (pixels where the face, etc. is imaged). That is, the user area is an area in the image data where all or part of the user's body is shown.

[0049] In the example of FIG. 6, the server device 10 generates image data as shown in FIGS. 7A to 7C. FIG. 7A shows an example of image data taken at time t1 and subjected to privacy protection processing. FIG. 7B shows an example of image data taken at time t2 and subjected to privacy protection processing. FIG. 7C shows an example of image data taken at time t3 and subjected to privacy protection processing.

[0050] As shown in FIGS. 7A to 7C, the user area of the person 41 for whom the user information notification has been sent from the terminal 30 to the camera device 20 is mosaicked, and the person 42 for whom the user information notification has not been sent is shown as it is in the image data.

[0051] The server device 10 stores the image data with privacy protection processing applied to the user area and the information of the user whose privacy is protected, in association with each other. Specifically, the server device 10 stores in association an image ID for identifying the image data, the date and time when the image data was acquired, the user ID of the user whose privacy is protected, the user position, and the attribute information.

[0052] For example, in the example of FIG. 7A, the server device 10 stores in association information such as image ID = pID01, time t1, user ID = uID01, user position (P1, Q1), and a 20-year-old female.

[0053] In this way, when the first image data (the original image data captured by the camera device 20) includes a user area corresponding to a user, the server device 10 performs privacy protection processing for protecting the privacy of the user on the user area. The server device 10 generates second image data (image data with the user's privacy protected) by performing the privacy protection processing, and stores the generated second image data.

[0054] Subsequently, details of each device included in the information processing system according to the first embodiment will be described.

[0055] [Terminal] The terminal 30 is exemplified by a mobile terminal device such as a smartphone, a mobile phone, a game machine, a tablet, etc. The terminal 30 can be any device or apparatus as long as it can receive the operation of the user and communicate with the server device 10 and the camera device 20.

[0056] FIG. 8 is a diagram showing an example of the processing configuration (processing module) of the terminal 30 according to the first embodiment. Referring to FIG. 8, the terminal 30 includes a communication control unit 201, an attribute information generation unit 202, a user information notification unit 203, and a storage unit 204.

[0057] The function related to the user's privacy protection is realized by a privacy mask application installed in the terminal 30. That is, the attribute information generation unit 202, the user information notification unit 203, etc. are realized by the application.

[0058] The communication control unit 201 is a means for controlling communication with other devices. For example, the communication control unit 201 receives data (packets) from a server on the network. Also, the communication control unit 201 transmits data to the server. The communication control unit 201 delivers the data received from other devices to other processing modules. The communication control unit 201 transmits the data acquired from other processing modules to other devices. In this way, other processing modules perform data transmission and reception with other devices via the communication control unit 201. The communication control unit 201 has a function as a receiving unit for receiving data from other devices and a function as a transmitting unit for transmitting data to other devices.

[0059] Also, the communication control unit 201 supports short-range wireless communication such as Bluetooth (registered trademark), and communicates with the camera device 20 by the proximity wireless communication. When the connection with the camera device 20 is completed, the communication control unit 201 notifies the user information notification unit 203 to that effect. It is assumed that the pairing process required when the terminal 30 and the camera device 20 communicate by Bluetooth (registered trademark) has been completed in advance.

[0060] The attribute information generation unit 202 is a means for generating the user's attribute information. The attribute information generation unit 202 acquires the biometric information (e.g., face image) of the user and a copy of the identity verification document (image data obtained by photographing the identity verification document) according to the user's operation.

[0061] The attribute information generation unit 202 acquires the biometric information of the user and a copy of the identity verification document using a GUI (Graphical User Interface) as shown in FIG. 9. In FIG. 9, while visually recognizing their own face captured by the terminal 30, the user presses the "Shoot" button at an appropriate timing. The attribute information generation unit 202 acquires an image including the face area of the user in response to the pressing of the button.

[0062] The attribute information generation unit 202 extracts a face image from the acquired image. Note that since existing technologies can be used for the face image detection process and face image extraction process by the attribute information generation unit 202, detailed descriptions are omitted. For example, the attribute information generation unit 202 may extract a face image (face area) from the image data using a learning model learned by a CNN (Convolutional Neural Network). Alternatively, the attribute information generation unit 202 may extract a face image using a method such as template matching.

[0063] Furthermore, the attribute information generation unit 202 acquires the face image described in the identity verification document (extracts the face image from the identity verification document).

[0064] The attribute information generation unit 202 executes an authentication process (one-to-one authentication) using two face images (the face image obtained by self-photographing and the face image extracted from the identity verification document). Specifically, the attribute information generation unit 202 generates feature amounts from each of the two face images.

[0065] Regarding the generation process of feature quantities, existing technologies can be used, so detailed descriptions thereof are omitted. For example, the attribute information generation unit 202 extracts eyes, nose, mouth, etc. from a face image as feature points. Then, the attribute information generation unit 202 calculates the positions of the respective feature points and the distances between the feature points as feature quantities, and generates a feature vector (vector information characterizing the face image) composed of a plurality of feature quantities.

[0066] The attribute information generation unit 202 calculates the similarity between the two generated feature quantities. For the similarity, the chi-square distance, Euclidean distance, etc. can be used. Note that the greater the distance, the lower the similarity, and the closer the distance, the higher the similarity.

[0067] If the similarity is equal to or greater than a predetermined value, the attribute information generation unit 202 determines that the personal verification (authentication) has succeeded. If the similarity is less than the predetermined value, the attribute information generation unit 202 determines that the personal verification has failed.

[0068] When the personal verification is successful, the attribute information generation unit 202 reads attribute information (gender, date of birth, etc.) from the identity verification document. Specifically, the attribute information generation unit 202 reads gender, date of birth, etc. from the identity verification document by OCR (Optical Character Recognition) technology. The attribute information generation unit 202 calculates the age of the user from the date of birth.

[0069] After that, the attribute information generation unit 202 generates a user ID for identifying the user. The user ID may be any information as long as it can uniquely identify the user. For example, the attribute information generation unit 202 may use a passport number, etc. obtained from the identity verification document as the user ID, or may use a hash value generated from a passport number, etc. as the user ID.

[0070] The attribute information generation unit 202 stores the user ID and attribute information (person annotation; for example, gender, age, or a combination thereof) in the storage unit 204.

[0071] In this way, when the attribute information generation unit 202 succeeds in one-to-one authentication using the user's first biometric information and the second biometric information described in the user's identity document, it acquires the attribute information from the identity document. Note that when the attribute information generation unit 202 succeeds in one-to-one authentication, it becomes possible to send a user information notification. That is, user privacy protection is not performed unless the user's attribute information has been acquired.

[0072] The user information notification unit 203 is a means for notifying the camera device 20 of the user's user information (user ID, attribute information, date and time information, and terminal location).

[0073] When the terminal 30 and the camera device 20 are connected, the user information notification unit 203 reads out the user ID and attribute information stored in the storage unit 204.

[0074] In addition, the user information notification unit 203 generates the position information of its own device. Specifically, the user information notification unit 203 receives GPS signals from GPS (Global Positioning System) satellites and performs positioning to generate position information (terminal location) including the latitude and longitude of the terminal 30. Alternatively, the user information notification unit 203 may generate position information based on the intensity of radio waves received from a wireless access point.

[0075] The user information notification unit 203 sends a user information notification including the user ID, attribute information, date and time information (current time), and terminal location to the camera device 20. Note that the user information notification unit 203 may not send the date and time information (current time) to the camera device 20 if necessary.

[0076] While communicating with the same camera device 20, the user information notification unit 203 sends the above user information notification to the camera device 20 at regular or predetermined timings. For example, the user information notification unit 203 sends the user information notification to the camera device 20 every second.

[0077] The storage unit 204 is a means for storing information necessary for the operation of the terminal 30. For example, the storage unit 204 stores the user ID, attribute information, and the like.

[0078] [Camera device] The camera device 20 is a photographing device (imaging device) that photographs a predetermined photographing area.

[0079] FIG. 10 is a diagram showing an example of the processing configuration (processing module) of the camera device 20 according to the first embodiment. Referring to FIG. 10, the camera device 20 includes a communication control unit 301, a user information control unit 302, and a storage unit 303.

[0080] The communication control unit 301 is a means for controlling communication with other devices. For example, the communication control unit 301 receives data (packets) from the server device 10. Also, the communication control unit 301 transmits data to the server device 10. The communication control unit 301 delivers the data received from other devices to other processing modules. The communication control unit 301 transmits the data acquired from other processing modules to other devices. In this way, other processing modules perform data transmission and reception with other devices via the communication control unit 301. The communication control unit 301 has a function as a receiving unit for receiving data from other devices and a function as a transmitting unit for transmitting data to other devices.

[0081] Also, the communication control unit 301 supports short-range wireless communication such as Bluetooth (registered trademark), and communicates with the terminal 30 by the proximity wireless communication. It is assumed that the pairing process necessary when the terminal 30 and the camera device 20 communicate via Bluetooth (registered trademark) has been completed in advance.

[0082] The user information control unit 302 is a means for controlling the control related to the user information notification acquired from the terminal 30. The user information control unit 302 receives the user information notification from the terminal 30. When the user information control unit 302 receives the user information notification, it controls the camera module (a module composed of a lens, an imaging element, etc.) and photographs the imaging area. That is, in response to the reception of the user information notification, the user information control unit 302 instructs the camera module to photograph the imaging area.

[0083] The user information control unit 302 notifies the server device 10 of the camera ID, the acquired image data, and the user information (user ID, attribute information, terminal location, etc.) included in the user information notification. Specifically, the user information control unit 302 transmits an "image data storage request" including the camera ID, the acquired image data, and the user information to the server device 10.

[0084] Note that if there is no instruction from the user information control unit 302, the camera module photographs the imaging area periodically (for example, several times per second). The camera device 20 periodically transmits the image data acquired together with the camera ID to the server device 10. The camera device 20 transmits an image data storage request including the camera ID and the image data to the server device 10.

[0085] The storage unit 303 is a means for storing information necessary for the operation of the camera device 20.

[0086] [Server device] FIG. 11 is a diagram showing an example of the processing configuration (processing module) of the server device 10 according to the first embodiment. Referring to FIG. 11, the server device 10 includes a communication control unit 401, an image data control unit 402, and a storage unit 403.

[0087] The communication control unit 401 is a means for controlling communication with other devices. For example, the communication control unit 401 receives data (packets) from the camera device 20. Also, the communication control unit 401 transmits data to the camera device 20. The communication control unit 401 delivers the data received from other devices to other processing modules. The communication control unit 401 transmits the data acquired from other processing modules to other devices. In this way, other processing modules perform data transmission and reception with other devices via the communication control unit 401. The communication control unit 401 has a function as a receiving unit that receives data from other devices and a function as a transmitting unit that transmits data to other devices.

[0088] The image data control unit 402 is a means for controlling the image data acquired from the camera device 20. FIG. 12 is a flowchart showing an example of the operation of the image data control unit 402 according to the first embodiment. The operation of the image data control unit 402 will be described while referring to FIG. 12.

[0089] The image data control unit 402 receives an "image data storage request" from the camera device 20. The image data control unit 402 identifies the transmission source of the image data (image data storage request) based on the camera ID included in the image data storage request (step S101).

[0090] The image data control unit 402 determines whether user information is included in the image data storage request (step S102). If user information is not included in the image data storage request (step S102, No branch), the image data control unit 402 stores the image data in the storage unit 403 (step S103).

[0091] Specifically, the image data control unit 402 generates an image ID corresponding to the acquired image data. Further, the image data control unit 402 registers the image ID, the acquisition date and time of the image data (current time), and the storage location of the image data (storage address) in the image information database (see FIG. 13).

[0092] The image information database is a database that stores information regarding the image data acquired from the camera device 20. Note that the image information database shown in FIG. 13 is an example and is not intended to limit the items to be stored. For example, the installation location of the camera device 20 derived from the camera ID may be stored in the image information database. Also, the image information database may be prepared for each camera device 20, or information regarding the image data acquired from each of a plurality of camera devices 20 may be stored in one image information database.

[0093] If the user information is included in the image data storage request (branch to Yes in step S102), the image data control unit 402 determines whether the user area of the user corresponding to the user information is included in the image data (step S104). That is, the image data control unit 402 determines whether the holder of the terminal 30 that has transmitted the user information notification is being photographed by the camera device 20.

[0094] Specifically, the image data control unit 402 determines whether the user area related to the holder of the terminal 30 is included in the image data based on the position information (position information of the terminal 30; terminal position) included in the user information.

[0095] Here, the area that the camera device 20 can photograph is determined in advance, and the range that is photographed as an image within the actual coordinate system (latitude, longitude) is also determined in advance. Therefore, the system administrator registers in advance in the server device 10 the coordinate range (range in the actual coordinate system) of the photographing area that is photographed as an image for each camera device 20 (for each camera ID).

[0096] The image data control unit 402 determines whether the terminal position (current position of the terminal 30) of the user information is included in the coordinate range corresponding to the image data obtained by the camera device 20 photographing the photographing area, and determines whether the user area is included in the image data.

[0097] That is, when the terminal 30 and the camera device 20 are in a communicable state via Bluetooth (registered trademark) or the like and the user information notification is transmitted to the camera device 20, it is determined whether or not the user has reached the shooting area of the camera device 20.

[0098] If the user area is not included in the image data (branch at step S104, No), the image data control unit 402 stores the acquired image data in the storage unit 403 in the same manner as normal image data (step S103).

[0099] If the user area is included in the image data (branch at step S104, Yes), the image data control unit 402 specifies the user area in the image data (step S105). Specifically, the image data control unit 402 converts the position information (X, Y) of the terminal 30 into the user position (P, Q) in the image coordinate system. The image data control unit 402 performs the above conversion using a coordinate conversion table or the like previously input to the server device 10 by a system administrator or the like.

[0100] The image data control unit 402 extracts an image area within a predetermined range centered on the user position (P, Q) in the image coordinate system. The image data control unit 402 inputs the extracted image area into the learning model to specify the user area in which a person appears.

[0101] Note that the learning model used by the image data control unit 402 is generated by machine learning using teacher data with labels (indicating the user area) attached to the images. Any algorithm such as a support vector machine, boosting, or neural network can be used for generating the learning model. Since algorithms such as the support vector machine are well-known techniques, their descriptions are omitted.

[0102] For example, as shown in FIG. 14A, the image data control unit 402 identifies the user position (P, Q) of the user who holds the terminal 30 from the terminal position (X, Y) of the terminal 30. Thereafter, the image data control unit 402 inputs the image area indicated by the dotted line in FIG. 14B to the learning model to identify the user area indicated by the dotted line.

[0103] The image data control unit 402 performs image processing for protecting the privacy of the user on the identified user area (executing privacy protection processing; step S106).

[0104] For example, the image data control unit 402 executes privacy protection processing such as applying a mosaic to the user area or filling in the user area.

[0105] The image data control unit 402 stores the image data after the privacy protection processing in the storage unit 403 (step S107). At this time, the image data control unit 402 registers information regarding the user whose privacy is protected (user ID, user position, attribute information) in the image information database in addition to the image ID of the image data, the acquisition date and time, and the storage location of the image data.

[0106] The storage unit 403 is a means for storing information necessary for the operation of the server device 10. The image information database is constructed in the storage unit 403. The storage unit 403 stores at least the second image data (image data after privacy protection processing), the user position of the user whose privacy is protected, and the attribute information in association with each other. Further, the storage unit 403 stores the second image data, the user position, the attribute information, and the user ID in association with each other. The storage unit 403 stores the image data and other information (such as the user position) in association with each other via the image ID.

[0107] [Operation of the system] Subsequently, the operation of the information processing system according to the first embodiment will be described.

[0108] FIG. 15 is a sequence diagram showing an example of the operation of the information processing system according to the first embodiment. Referring to FIG. 15, the operation of the information processing system regarding the privacy protection of the user will be described.

[0109] When the user approaches the camera device 20, the terminal 30 transmits a user information notification regarding the user to the camera device 20 (step S01).

[0110] In response to receiving the user information notification, the camera device 20 captures the shooting area and acquires image data (step S02).

[0111] The camera device 20 transmits an image data storage request including the acquired image data and user information, etc. to the server device 10 (step S03).

[0112] The server device 10 identifies the user area in which the user (the user who desires privacy protection) appears from the image data (step S04). The server device 10 identifies the user area in the image data based on the terminal position indicated by the latitude and longitude coordinate system of the terminal 30 included in the user information.

[0113] The server device 10 executes a privacy protection process on the identified user area (step S05). For example, the privacy protection process is a process of applying a mosaic or filling in the user area.

[0114] As described above, in the information processing system according to the first embodiment, the terminal 30 provides information about the user (user information, for example, attribute information) to the server device 10 via the camera device 20. When the server device 10 determines that the user who provides the attribute information is being photographed by the camera device 20, it executes a process for protecting the privacy of the user. Specifically, the server device 10 applies a mosaic or fills in the user area so that the user shown in the image data cannot be identified (specified) (applies a privacy mask to the user area). In this way, since the image of the person being photographed is masked (hidden) in response to providing the user information, the privacy of the person being photographed is appropriately protected. Also, the entity that installs the camera device 20 can grasp the general outline (portrait) of the user photographed by the camera device 20 based on the attribute information. That is, since sufficient image data for crime prevention and disaster prevention purposes can be obtained, there is no hindrance to the work of the entity that installs the camera device 20, etc.

[0115] [Second Embodiment] Subsequently, the second embodiment will be described in detail with reference to the drawings.

[0116] In the first embodiment, it was explained that the privacy of a user who notifies his or her own attributes (such as gender and age) to the camera device 20 is protected by masking the image of the user. That is, it is not possible to identify the user from the image data (still image, moving image) to which such privacy protection processing has been applied.

[0117] Here, the image captured by the camera device 20 is not only used by the entity that installed the camera device 20, but may also be used by an investigative agency such as the police. However, it is not possible to identify the user (person being photographed) from the image to which privacy protection processing has been applied, which may hinder the criminal investigation, etc. of the investigative agency.

[0118] In the second embodiment, the case where an investigative agency or the like can identify the identity of a user whose privacy is protected will be described.

[0119] Hereinafter, the differences between the first embodiment and the second embodiment will be mainly described.

[0120] FIG. 16 is a diagram showing an example of the schematic configuration of the information processing system according to the second embodiment. Referring to FIG. 16, a management server 40 is added to the configuration of the information processing system according to the first embodiment.

[0121] The management server 40 is a server that manages the identity information of users who wish to protect their privacy. The management server 40 stores the user ID and the identity information related to the identity of the user in association with each other. The management server 40 is operated and managed by a public agency such as the police or a company commissioned by a public agency. More specifically, the management server 40 is operated and managed by an organization, group, etc. different from the installation entity of the camera device 20.

[0122] When the user's terminal 30 (privacy mask application) acquires the user's attribute information, it transmits the user ID and identity information (for example, name, gender, age, address, etc.) of the user to the management server 40. More specifically, the terminal 30 transmits an identity information registration request including the user ID and identity information to the management server 40 (see FIG. 17).

[0123] In the process of investigating an incident or the like, the investigating agency requests the installation entity of the camera device 20 (hereinafter referred to as the camera administrator) to provide the image data (still images, videos) captured by the camera device 20. Specifically, an investigator or the like requests the camera administrator to view the image data by specifying the camera device 20 and the period (start date and time, end date and time; hereinafter referred to as the disclosure request period) for which information is requested.

[0124] The requested camera administrator (the installation entity of the camera device 20) operates the administrator terminal 50 and inputs the camera ID and date and time (time zone) of the camera device 20 for which viewing is requested into the server device 10. Specifically, the administrator terminal 50 transmits an "image data search request" including the camera ID and the search period to the server device 10 (see FIG. 18).

[0125] The server device 10 reads out from the storage unit 403 the image data corresponding to the acquired camera ID and the search period (start date / time and end date / time of the search), and transmits a response including the read-out image data to the administrator terminal 50. At this time, if the image data includes a user whose privacy is protected (a user with a mosaic or the like), the server device 10 also notifies the administrator terminal 50 of the user ID and the user position of the user.

[0126] For example, the server device 10 transmits a list consisting of a combination of the image ID, the user ID, and the user position to the administrator terminal 50. Alternatively, the server device 10 transmits the image data in which the user ID corresponding to the user area (the area to which the privacy protection process is applied) is written to the administrator terminal 50. For example, the server device 10 transmits the image data as shown in FIG. 19 to the administrator terminal 50.

[0127] The camera administrator provides the image data (for example, at least one or more image data as shown in FIG. 19) acquired from the server device 10 to the investigating agency. Specifically, the camera administrator operates the administrator terminal 50 to store the image data acquired from the server device 10 in an external storage device such as a USB (Universal Serial Bus) memory. The administrator or the like submits the external storage device to the investigating agency or the like. Alternatively, the camera administrator may provide the image data to the investigating agency by means such as e-mail.

[0128] The investigator or the like operates the investigating agency terminal 60 to visually confirm (check) the image data provided by the camera administrator. When the investigator or the like discovers a user whose privacy is protected and whose identity is to be confirmed, the investigator or the like inputs the user ID of the user to the management server 40. Specifically, the investigating agency terminal 60 transmits a request for providing identity information including the user ID to the management server 40 (see FIG. 20).

[0129] The management server 40 transmits a response including the identity information corresponding to the acquired user ID to the investigating agency terminal 60.

[0130] In this way, the server device 10 acquires search conditions regarding the search for stored image data from the first external device (administrator terminal 50). The server device 10 extracts image data that matches the search conditions and transmits the extracted image data to the first external device. The management server 40 receives an identity information provision request including a user ID from the second external device (investigation agency terminal 60) and transmits the identity information corresponding to the user ID to the second external device.

[0131] Subsequently, details of each device included in the information processing system of the second embodiment will be described.

[0132] [Terminal] FIG. 21 is a diagram showing an example of the processing configuration (processing module) of the terminal 30 according to the second embodiment. Referring to FIG. 21, an identity information provision unit 205 is added to the configuration of the terminal 30 according to the first embodiment.

[0133] The identity information provision unit 205 is a means for providing the management server 40 with the user's identity information (for example, name, gender, date of birth, address, etc.). The identity information provision unit 205 acquires identity information from the user's identity document (for example, passport, etc.). More specifically, the identity information provision unit 205 reads the identity information from the identity document using OCR technology or the like.

[0134] When the attribute information generation unit 202 generates the user's user ID, the identity information provision unit 205 transmits an identity information registration request including the user ID and the identity information to the management server 40.

[0135] [Administrator Terminal] FIG. 22 is a diagram showing an example of the processing configuration (processing module) of the administrator terminal 50 according to the second embodiment. Referring to FIG. 22, the administrator terminal 50 includes a communication control unit 501, a search request unit 502, and a storage unit 503.

[0136] The communication control unit 501 is a means for controlling communication with other devices. For example, the communication control unit 501 receives data (packets) from the server device 10. Also, the communication control unit 501 transmits data toward the server device 10. The communication control unit 501 delivers the data received from other devices to other processing modules. The communication control unit 501 transmits the data acquired from other processing modules toward other devices. In this way, other processing modules perform data transmission and reception with other devices via the communication control unit 501. The communication control unit 501 has a function as a receiving unit for receiving data from other devices and a function as a transmitting unit for transmitting data toward other devices.

[0137] The search request unit 502 is a means for requesting the server device 10 to search for image data. The search request unit 502 displays a GUI (Graphical User Interface) or the like for inputting details of the image data to be acquired from the server device 10 in response to an operation by the camera administrator (the entity that installed the camera device 20). That is, the search request unit 502 acquires the conditions (search conditions) for the server device 10 to search for image data from the camera administrator or the like.

[0138] For example, the search request unit 502 uses a GUI as shown in FIG. 23 to acquire the camera ID of the camera device 20 that captured the image data to be the target of the provision request (search target) and the search period. The search request unit 502 transmits an image data search request including the acquired camera ID and search period to the server device 10.

[0139] In this way, the search request unit 502 acquires the camera ID of the camera device 20 that captured the image desired by the investigator or the like and the shooting time zone, and transmits this information to the server device 10. Note that the camera administrator may refer to a list associating the installation locations of the camera devices 20 included in the system with the camera IDs of each camera device 20 to specify the camera ID of the camera device 20 that the investigator or the like wishes to view.

[0140] The search request unit 502 receives a response (positive response, negative response) to the image data search request from the server device 10.

[0141] When a positive response is received, the search request unit 502 notifies the camera administrator to that effect. Further, the search request unit 502 displays the acquired image data or stores the image data in a USB memory according to the operation of the camera administrator.

[0142] When a negative response is received, the search request unit 502 notifies the camera administrator that the image data corresponding to the search conditions (camera ID, search period) is not stored.

[0143] The storage unit 503 is a means for storing information necessary for the operation of the administrator terminal 50.

[0144] [Server device] FIG. 24 is a diagram showing an example of the processing configuration (processing module) of the server device 10 according to the second embodiment. Referring to FIG. 24, a search request processing unit 404 is added to the configuration of the server device 10 according to the second embodiment.

[0145] The search request processing unit 404 is a means for processing the image data search request received from the administrator terminal 50. The search request processing unit 404 accesses the image information database and extracts an entry corresponding to the search conditions (camera ID, search period) included in the image data search request.

[0146] The search request processing unit 404 refers to the storage location field of the extracted entry and reads the corresponding image data from the storage unit 403. The search request processing unit 404 transmits a positive response including the read image data (image data with an image ID) to the administrator terminal 50. When an entry that matches the search conditions does not exist in the image information database, the search request processing unit 404 transmits a negative response to the administrator terminal 50 as a response to the image data search request.

[0147] Here, when the read image data includes image data to which privacy protection processing has been applied, the search request processing unit 404 notifies the administrator terminal 50 of information for identifying the user to whom the privacy protection processing has been applied. Specifically, the search request processing unit 404 notifies the administrator terminal 50 of the user ID and the user location of the user to whom the privacy protection processing has been applied.

[0148] At that time, the search request processing unit 404 may transmit to the administrator terminal 50 a list associating the image ID, the user ID, and the user location of the image data to which the privacy protection processing has been performed, together with the image data that matches the search conditions. The search request processing unit 404 may transmit an affirmative response including the image data and the above list to the administrator terminal 50.

[0149] Alternatively, the search request processing unit 404 may identify the user area in the image data where mask processing or the like has been performed based on the set value (user location) of the user location field of the extracted entry, and write the user ID near the identified area. Specifically, the search request processing unit 404 may generate image data as shown in FIG. 19. The search request processing unit 404 may notify the administrator terminal 50 of the user ID and the user location of the user whose privacy has been protected by transmitting the image data with the user ID written therein to the administrator terminal 50.

[0150] In this way, the server device 10 determines whether the image data extracted as the image data that matches the search conditions includes image data (second image data) to which privacy protection processing has been applied. When the second image data is included, the server device 10 notifies the external device (administrator terminal 50) of the user ID and the user location corresponding to the user whose privacy has been protected in the extracted second image data. At that time, the server device 10 may generate third image data by writing the user ID at a location corresponding to the user location of the extracted second image data, and transmit the generated third image data to the external device.

[0151] [Management Server] FIG. 25 is a diagram showing an example of the processing configuration (processing modules) of the management server 40 according to the second embodiment. Referring to FIG. 25, the management server 40 includes a communication control unit 601, an identity information registration control unit 602, an identity information provision control unit 603, and a storage unit 604.

[0152] The communication control unit 601 is means for controlling communication with other devices. For example, the communication control unit 601 receives data (packets) from the investigative agency terminal 60. Also, the communication control unit 601 transmits data to the investigative agency terminal 60. The communication control unit 601 delivers the data received from other devices to other processing modules. The communication control unit 601 transmits the data acquired from other processing modules to other devices. In this way, other processing modules perform data transmission and reception with other devices via the communication control unit 601. The communication control unit 601 has a function as a receiving unit for receiving data from other devices and a function as a transmitting unit for transmitting data to other devices.

[0153] The identity information registration control unit 602 is means for controlling registration regarding the identity information of users. For example, the identity information registration control unit 602 receives an identity information registration request from the user terminal 30 and acquires the user ID and identity information of the user.

[0154] The identity information registration control unit 602 associates the user ID and the identity information and stores them in the user information database (see FIG. 26). Note that the user information database shown in FIG. 26 is an example and is not intended to limit the items to be stored. For example, a "face image" may be registered in the user information database as biometric information.

[0155] The identity information provision control unit 603 is means for processing an identity information provision request from the investigative agency terminal 60. When receiving an identity information provision request, the identity information provision control unit 603 searches the user information database using the user ID included in the request as a key and identifies the corresponding identity information.

[0156] If there is identity information corresponding to the user ID, the identity information providing control unit 603 transmits an affirmative response including the identity information to the investigative agency terminal 60. If there is no identity information corresponding to the user ID, the identity information providing control unit 603 transmits a negative response indicating that fact to the investigative agency terminal 60.

[0157] The storage unit 604 is a means for storing information necessary for the operation of the management server 40. The user information database is configured in the storage unit 604.

[0158] [Investigative Agency Terminal] Examples of the investigative agency terminal 60 include mobile terminal devices such as smartphones, mobile phones, game machines, and tablets, and computers (personal computers, notebook computers), etc. The investigative agency terminal 60 can be any device or apparatus as long as it can receive operations from investigators or the like and communicate with the management server 40 or the like. Also, since the configuration of the investigative agency terminal 60 and the like are obvious to those skilled in the art, detailed descriptions are omitted.

[0159] As described above, in the second embodiment, the management server 40 manages (stores) the identity information of users who desire privacy protection. The investigative agency acquires the image data to which the privacy protection process has been applied and the user ID of the user whose privacy has been protected from the camera administrator, and can obtain the corresponding identity information by transmitting the user ID to the management server 40. That is, even for a user whose privacy is protected by providing attribute information (for example, gender, age, etc.) as described in the first embodiment, if the investigative agency determines it is necessary, the identity information of the user will be provided to the investigative agency. As a result, even if the privacy protection process is executed, it will not hinder the criminal investigation or the like by the investigative agency.

[0160] Subsequently, the hardware of each device constituting the information processing system will be described. FIG. 27 is a diagram showing an example of the hardware configuration of the server device 10.

[0161] The server device 10 can be configured by an information processing device (so-called computer) and has the configuration illustrated in FIG. 27. For example, the server device 10 includes a processor 311, a memory 312, an input / output interface 313, a communication interface 314, and the like. The components such as the processor 311 are connected by an internal bus or the like and are configured to be able to communicate with each other.

[0162] However, the configuration shown in FIG. 27 is not intended to limit the hardware configuration of the server device 10. The server device 10 may include hardware not shown, or may not include the input / output interface 313 as necessary. Also, the number of components such as the processor 311 included in the server device 10 is not intended to be limited to the example shown in FIG. 27. For example, a plurality of processors 311 may be included in the server device 10.

[0163] The processor 311 is a programmable device such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a DSP (Digital Signal Processor), for example. Alternatively, the processor 311 may be a device such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The processor 311 executes various programs including an operating system (OS; Operating System).

[0164] The memory 312 is a RAM (Random Access Memory), a ROM (Read Only Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), or the like. The memory 312 stores an OS program, an application program, and various data.

[0165] The input / output interface 313 is an interface for a display device and an input device (not shown). The display device is, for example, a liquid crystal display or the like. The input device is a device that receives user operations such as a keyboard and a mouse.

[0166] The communication interface 314 is a circuit, a module, etc. that communicate with other devices. For example, the communication interface 314 includes a NIC (Network Interface Card) or the like.

[0167] The functions of the server device 10 are realized by various processing modules. The processing modules are realized, for example, when the processor 311 executes a program stored in the memory 312. Also, the program can be recorded on a computer-readable storage medium. The storage medium can be non-transitory such as a semiconductor memory, a hard disk, a magnetic recording medium, an optical recording medium, etc. That is, the present invention can also be embodied as a computer program product. Also, the above program can be downloaded via a network or updated using a storage medium storing the program. Further, the above processing module may be realized by a semiconductor chip.

[0168] Note that a management server 40 or the like can also be configured by an information processing device in the same manner as the server device 10, and its basic hardware configuration is the same as that of the server device 10, so the description is omitted. Also, the camera device 20 only needs to include a camera module or the like for photographing a user.

[0169] The server device 10, which is an information processing device, is equipped with a computer, and the functions of the server device 10 can be realized by causing the computer to execute a program. Also, the server device 10 executes a control method of the server device 10 according to the program.

[0170] [Modification Example] Note that the configuration, operation, etc. of the information processing system described in the above embodiment are examples, and are not intended to limit the configuration of the system or the like.

[0171] In the above embodiment, when attribute information (person annotation) is provided, in order to maximize the protection of the privacy of the user who provides the attribute information, it has been described that a highly confidential privacy protection process is executed. However, this fact does not prevent the protection of the privacy of users who do not provide attribute information. The server device 10 may apply a privacy protection process to the user area of a user who does not provide attribute information as needed. That is, the server device 10 may apply a privacy protection process in line with the purpose of collecting image data to the user area of the above user. At that time, the server device 10 may change the content of the privacy protection process between normal users and users who provide attribute information. For example, the server device 10 may apply mosaics to the user area of normal users and may fill in the user area of users who provide attribute information.

[0172] The server device 10 may apply privacy protection processes with different contents for each part (face, body) even in the user area of the same person. For example, mosaics may be applied to the face area and the body area may be filled in, etc.

[0173] When the server device 10 stores the user's attribute information, it may notify the user to that effect. Specifically, the terminal 30 transmits a user information notification including a contact such as an email address to the server device 10 via the camera device 20. When the server device 10 applies a privacy protection process to the user area of the user and stores the image data to which the privacy protection process has been applied and the attribute information, it may notify the contact to that effect.

[0174] Alternatively, when the image data to which the privacy protection process has been applied is deleted due to the expiration of the storage period or the like, the server device 10 may notify the user (terminal 30) to that effect. Further, when the image data to which the privacy protection process has been applied is deleted, the server device 10 may also delete the information of the user (user ID, attribute information, etc.) whose privacy was protected by the said image data.

[0175] The server device 10 may transmit a notification to the effect that the above attribute information has been stored or a notification to the effect that the image data has been deleted in response to the user's request to receive a user information notification from the user's terminal 30. Specifically, the fact that the above attribute information has been stored may be notified to the terminal 30 at the timing of the transmission of the user information notification from the terminal 30 shown in FIG. 5.

[0176] When executing the privacy protection process, the server device 10 may change the content of the privacy protection process according to the user's attribute information. For example, the server device 10 may change the content of the privacy protection process according to gender or age. For example, the server device 10 may change the color or pattern when filling the user area according to gender or age. That is, the server device 10 may change the privacy mask given to the user according to the user's attribute information (see FIGS. 28A and 28B). In FIG. 28A, a privacy mask of horizontal lines is given to a male user. In FIG. 28B, a privacy mask of fine mosaic for the eyes is given to a female user. Thus, the server device 10 may apply a privacy mask with different coarseness of mosaic or a privacy mask with a changed pattern (diagonal lines, horizontal lines) to the user area for each user attribute (age, gender).

[0177] When executing the privacy protection process, the server device 10 may replace the user (user area) with a character or an avatar. Further, the server device 10 may select a character or an avatar to be replaced according to the user's attribute information (gender, age). Also, the terminal 30 may transmit the image data of the character or avatar used during the execution of the privacy protection process to the camera device 20 (server device 10) as user information.

[0178] The server device 10 may write the user's attribute information to the image data to which the privacy protection process is applied. For example, the server device 10 may store image data as shown in FIG. 29.

[0179] The server device 10 may determine whether to execute the privacy protection process and the content of the privacy protection process based on a predetermined policy. For example, the above policy may include contents such as "Do not apply the privacy protection process to normal users (users who do not provide attribute information)", "Apply avatar B1 to female users and apply avatar B2 to male users". Alternatively, the above policy may be in the content such as "Apply the privacy protection process to users photographed during the day and do not apply the privacy protection process to users photographed at night". Regarding what kind of policy to set, the camera administrator may determine it in view of the installation purpose of the camera device 20 and the like.

[0180] In the above embodiment, it has been described that the terminal 30 executes the authentication process using biometric information (face image) and generates attribute information when the authentication is successful. However, the terminal 30 may generate the attribute information without executing the authentication process. For example, the terminal 30 may generate attribute information from the photographed identity document, or may generate attribute information based on the self-declaration from the user. That is, when the information processing system can allow false attribute information to be registered, user biometric authentication is unnecessary. In other words, in the above embodiment, in order to ensure the reliability of the attribute information, the attribute information is acquired from the identity document of the user who has successfully passed the biometric authentication.

[0181] In the above-described embodiment, it has been described that the server device 10 stores image data (image ID) to which privacy protection processing has been applied. Further, the server device 10 may store the image data to which privacy protection processing has been applied in association with the original image data. Also, in this case, instead of transmitting the image data to which privacy protection processing has been applied, the user ID, and the user location to the administrator terminal 50, the server device 10 according to the second embodiment may transmit the original image data to the administrator terminal 50. That is, the original image data may be provided to the investigative agency instead of the image data (video data) to which privacy protection processing has been applied.

[0182] In the above-described embodiment, the explanation has been made on the premise that the user who wishes to protect privacy starts the privacy mask application of the terminal 30. However, when the terminal 30 becomes communicable with the camera device 20 (when the user enters the communicable area of the short-range wireless communication), the terminal 30 may display a message prompting the start of the privacy mask application. For example, the terminal 30 may display a pop-up prompting the start of the privacy mask application, and the user may start the privacy mask application in response to the display.

[0183] The privacy mask application installed on the terminal 30 may allow the user to view the communication history with the camera device 20. In that case, the privacy mask application may display a communication history including the date and time, location, etc. when the user information was transmitted to the camera device 20.

[0184] In the above-described second embodiment, the server device 10 may transmit the attribute information of the user to the administrator terminal 50 instead of or in addition to the user ID. That is, the attribute information (person annotation) of the user may be provided to the investigative agency.

[0185] In the second embodiment, the identity information provision request transmitted by the investigative agency terminal 60 may include a plurality of user IDs. The investigative agency terminal 60 may transmit a plurality of user IDs obtained from different image data to the management server 40 in one identity information provision request.

[0186] In the second embodiment, the case where the investigative agency terminal 60 transmits an identity information provision request to the management server 40 has been described. However, the server device 10 may transmit an identity information provision request including a user ID to the management server 40. The server device 10 may transmit the identity information obtained from the management server 40 to the administrator terminal 50.

[0187] In the above embodiment, it has been described that the terminal 30 notifies the server device 10 of the current position of the user (terminal position of the terminal 30) via the camera device 20. However, the terminal 30 may directly notify the server device 10 of user information including the terminal position at regular intervals or at a predetermined timing. When the server device 10 determines that the user has entered the shooting area of the camera device 20 based on the terminal position obtained from the terminal 30 and the installation position of the camera device 20, the server device 10 may execute a process for protecting the privacy of the user. By such a measure, even when a camera device that does not support short-range wireless communication such as Bluetooth (registered trademark) is included in the system, the privacy of the user can be appropriately protected.

[0188] Alternatively, when it is difficult to identify the user to whom the privacy protection process is applied from the position information of the terminal 30 due to the accuracy of the position information measured by the terminal 30 or the like, the user to whom the privacy protection process is applied may be identified using biometric authentication. In this case, the user registers their biometric information (e.g., face image) and attribute information with the server device 10 in advance. The server device 10 extracts the face area from the image acquired from the camera device 20 and performs biometric authentication using the extracted face area (face image) and the face image registered in advance. When the user is identified by biometric authentication, the server device 10 applies the privacy protection process to the identified user. Also, by combining the identification of the user using the terminal position of the terminal 30 and the identification of the user using biometric information, the user may be identified with higher accuracy. For example, a rough area where the user appears may be identified based on the terminal position of the terminal 30, and the final user to whom the privacy protection process is applied may be identified by biometric authentication from within the identified rough area.

[0189] Note that, as described above, when the user's terminal 30 communicates directly with the server device 10 without going through the camera device 20, the user can be identified by biometric authentication and the privacy protection process can be applied to the user. In this regard, when the terminal 30 communicates with the camera device 20, the user can be identified by the following two methods. The first method is a method in which the terminal 30 transmits the user's biometric information to the camera device 20, and the camera device 20 transmits the captured image and the biometric information to the server device 10. The server device 10 identifies the user from the image by biometric authentication. By such a method, the server device 10 can associate the camera device 20 with the user (identify the user). The second method is a method in which the terminal 30 transmits the position information to the camera device 20, and the camera device 20 transmits the acquired position information, its own position information, and the captured image to the server device 10. The server device 10 identifies the user in the image based on the two position information. Also by such a method, the server device 10 can associate the camera device 20 with the user (identify the user).

[0190] The terminal 30 and the camera device 20 may communicate with each other by means other than Bluetooth (registered trademark). For example, the terminal 30 and the camera device 20 may communicate with each other by means of ZigBee (registered trademark) or the like. Alternatively, the terminal 30 and the camera device 20 may communicate with each other according to a standard corresponding to a wireless LAN (Local Area Network).

[0191] In the above embodiment, the case where the image information database is configured inside the server device 10 has been described. However, the database may be constructed in an external database server or the like. That is, some functions of the server device 10 may be implemented in another server.

[0192] In the above embodiment, the case where the server device 10 executes the privacy protection process has been described. However, the camera device 20 that has acquired the image data may execute the privacy protection process. The camera device 20 may transmit the image data to which the privacy protection process has been applied and the corresponding user information (user ID, user location, etc.) to the server device 10. In this way, the camera device 20 may have all or some of the functions of the server device 10.

[0193] In the above embodiment, the description has been made on the premise that the shooting area of the camera device 20 is fixed. However, the camera device 20 may be a camera (imaging device) whose shooting area changes with time.

[0194] The form of data transmission and reception between each device (server device 10, camera device 20, terminal 30, etc.) is not particularly limited, but the data transmitted and received between these devices may be encrypted. Between these devices, image data including the user's face image is transmitted and received, and in order to appropriately protect this information, it is desirable that encrypted data is transmitted and received.

[0195] In the flowcharts (flowcharts, sequence diagrams) used in the above description, a plurality of steps (processes) are described in order, but the execution order of the steps executed in the embodiments is not limited to the order of the description. In the embodiments, for example, the order of the illustrated steps can be changed within a range that does not substantially affect the content, such as executing each process in parallel.

[0196] The above embodiments have been described in detail for ease of understanding of the present disclosure, and it is not intended that all the configurations described above are necessary. Also, when a plurality of embodiments are described, each embodiment may be used alone or in combination. For example, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace a part of the configuration of one embodiment with another configuration.

[0197] From the above description, the industrial applicability of the present invention is clear, and the present invention is suitably applicable to an information processing system including a management camera and the like.

[0198] Some or all of the above embodiments may be described as follows in the appended claims, but are not limited thereto. [Appended Claim 1] A camera device that photographs a predetermined area, A terminal possessed by a user, A server device that stores the image data captured by the camera device in association with the acquisition date and time of the image data, including When the terminal becomes communicable with the camera device, it transmits a user information notification including user information regarding the user to the camera device, The camera device transmits the first image data obtained by photographing the predetermined area in response to the reception of the user information notification and the user information to the server device, When the user area corresponding to the user is included in the first image data, the server device generates second image data by performing privacy protection processing for protecting the privacy of the user with respect to the user area, and stores the generated second image data. [Appendix 2] The user information includes the attribute information of the user. The server device The system according to Appendix 1, wherein the second image data, the user position in the second image data of the user whose privacy is protected, and the attribute information are stored in association with each other. [Appendix 3] The user information further includes a user ID for identifying the user. The system according to Appendix 2, wherein the server device stores the second image data, the user position, the attribute information, and the user ID in association with each other. [Appendix 4] The system according to Appendix 3, wherein the server device obtains a search condition regarding a search for the stored image data from a first external device, extracts image data that matches the search condition, and transmits the extracted image data to the first external device. [Appendix 5] The system according to Appendix 4, wherein when the second image data is included in the image data extracted as the image data that matches the search condition, the server device notifies the first external device of the user ID and the user position corresponding to the user whose privacy is protected in the extracted second image data. [Appendix 6] The system according to Appendix 5, wherein the server device writes the user ID at a location corresponding to the user position of the extracted second image data to generate third image data, and transmits the generated third image data to the first external device. [Appendix 7] The system further includes a management server that stores the user ID and the identity information regarding the identity of the user in association with each other. The management server is the system according to Supplementary Note 6, which receives an identity information provision request including the user ID from a second external device and transmits the identity information corresponding to the user ID to the second external device. [Supplementary Note 8] The user information includes the terminal position indicated in the latitude and longitude coordinate system of the terminal. The server device is the system according to any one of Supplementary Notes 2 to 7, which specifies the user area in the image data based on the terminal position. [Supplementary Note 9] The terminal is the system according to any one of Supplementary Notes 2 to 8, which enables the transmission of the user information notification when the one-to-one authentication using the first biometric information of the user and the second biometric information described in the user's identity document is successful. [Supplementary Note 10] The terminal is the system according to Supplementary Note 9, which acquires the attribute information from the identity document when the one-to-one authentication is successful. [Supplementary Note 11] The attribute information is any one of the user's gender, age, and the combination of gender and age according to any one of Supplementary Notes 2 to 10. [Supplementary Note 12] The privacy protection process is the process of applying a mosaic or filling in the user area according to any one of Supplementary Notes 1 to 11. [Supplementary Note 13] A storage unit that associates and stores the image data captured by a camera device that can capture a predetermined area and can communicate with a terminal held by a user, and receives a user information notification including user information related to the user from the terminal, with the acquisition date and time of the image data. A receiving unit that receives the first image data obtained by the camera device capturing the predetermined area in response to the reception of the user information notification and the user information from the camera device. When the first image data includes a user area corresponding to the user, second image data is generated by performing a privacy protection process for protecting the privacy of the user with respect to the user area, and the generated second image data is stored, an image data control unit; A server device comprising the same. [Appendix 14] In a server device, When it is possible to photograph a predetermined area and communicate with a terminal possessed by a user, the image data photographed by a camera device that receives a user information notification including user information about the user from the terminal is stored in association with the acquisition date and time of the image data, The first image data obtained by the camera device photographing the predetermined area in response to the reception of the user information notification and the user information are received from the camera device, When the first image data includes a user area corresponding to the user, second image data is generated by performing a privacy protection process for protecting the privacy of the user with respect to the user area, and the generated second image data is stored, a control method of a server device. [Appendix 15] On a computer mounted on a server device, When it is possible to photograph a predetermined area and communicate with a terminal possessed by a user, a process of storing the image data photographed by a camera device that receives a user information notification including user information about the user from the terminal in association with the acquisition date and time of the image data, A process of receiving the first image data obtained by the camera device photographing the predetermined area in response to the reception of the user information notification and the user information from the camera device, When the first image data includes a user area corresponding to the user, a process of generating second image data by performing a privacy protection process for protecting the privacy of the user with respect to the user area and storing the generated second image data, A computer-readable storage medium storing a program for executing the same.

[0199] Note that the disclosures of the above-cited prior art documents are incorporated herein by reference. Having described the embodiments of the present invention above, the present invention is not limited to these embodiments. It will be understood by those skilled in the art that these embodiments are merely illustrative and that various modifications can be made without departing from the scope and spirit of the present invention. That is, the present invention naturally includes all disclosures including the claims, various modifications and corrections that can be made by those skilled in the art in accordance with the technical idea.

Explanation of Reference Numerals

[0200] 10 Server device 20 Camera device 20-1 Camera device 20-2 Camera device 30 Terminal 40 Management server 41 Person 42 Person 50 Administrator terminal 60 Search agency terminal 101 Camera device 102 Terminal 103 Server device 201 Communication control unit 202 Attribute information generation unit 203 User information notification unit 204 Storage unit 205 Identity information providing unit 301 Communication control unit 302 User information control unit 303 Storage unit 311 Processor 312 Memory 313 Input / output interface 314 Communication interface 401 Communication control unit 402 Image data control unit 403 Storage unit 404 Search request processing unit 501 Communication control unit 502 Search Requirement Section 503 Memory Section 601 Communication Control Section 602 Identity Information Registration Control Section 603 Identity Information Provision Control Section 604 Memory Section

Claims

1. A camera device that captures a predetermined area, A terminal possessed by a user, A server device that stores the image data captured by the camera device in association with the acquisition date and time of the image data, Including, When the terminal becomes communicable with the camera device, the terminal transmits a user information notification including user information regarding the user to the camera device, The camera device transmits the first image data obtained by capturing the predetermined area in response to receiving the user information notification and the user information to the server device, When the user area corresponding to the user is included in the first image data, the server device generates second image data by performing a privacy protection process for protecting the privacy of the user with respect to the user area, stores the generated second image data, A system that notifies the user that the second image data has been stored.

2. The server device notifies the user that the stored second image data has been deleted when the stored second image data is deleted. The system according to claim 1.

3. The user information includes the user's attribute information, The server device, Stores the second image data, the user position in the second image data of the user whose privacy is protected, and the attribute information in association with each other. The system according to claim 2.

4. The user information further includes a user ID for identifying the user, The server device stores the second image data, the user position, the attribute information, and the user ID in association with each other. The system according to claim 3.

5. When the server device acquires a search condition regarding the search for the stored image data from a first external device, extracts the image data that matches the search condition, and transmits the extracted image data to the first external device. The system according to claim 4.

6. When the second image data is included in the image data extracted as the image data that matches the search condition, the server device notifies the first external device of the user ID and the user position corresponding to the user whose privacy is protected in the extracted second image data. The system according to claim 5.

7. The server device writes the user ID to a location corresponding to the user position in the extracted second image data to generate third image data, and transmits the generated third image data to the first external device. The system according to claim 6.

8. Further comprising a management server that stores the user ID and identity information regarding the identity of the user in association with each other, The management server receives an identity information provision request including the user ID from a second external device, and transmits the identity information corresponding to the user ID to the second external device. The system according to claim 7.

9. A storage unit that stores, in association with each other, image data captured by a camera device that can capture a predetermined area and can communicate with a terminal possessed by a user, and receives a user information notification including user information regarding the user from the terminal, and the acquisition date and time of the image data; A receiving unit that receives the first image data obtained by the camera device capturing the predetermined area in response to the reception of the user information notification and the user information from the camera device; When the user area corresponding to the user is included in the first image data, a second image data is generated by performing a privacy protection process for protecting the privacy of the user with respect to the user area, the generated second image data is stored, and the user is notified that the second image data has been stored. An image data control unit; A server device comprising:

10. In a server device, A storage unit that stores, in association with each other, image data captured by a camera device that can capture a predetermined area and can communicate with a terminal possessed by a user, and receives a user information notification including user information regarding the user from the terminal, and the acquisition date and time of the image data; A receiving unit that receives the first image data obtained by the camera device capturing the predetermined area in response to the reception of the user information notification and the user information from the camera device; When the user area corresponding to the user is included in the first image data, a second image data is generated by performing a privacy protection process for protecting the privacy of the user with respect to the user area, and the generated second image data is stored; A control method for a server device that notifies the user that the second image data has been stored.

11. On a computer mounted on a server device, When a camera device capable of photographing a predetermined area and communicating with a terminal possessed by a user receives a user information notification including user information about the user from the terminal, the image data photographed by the camera device is stored in association with the acquisition date and time of the image data. A process of receiving the first image data obtained by photographing the predetermined area by the camera device in response to the reception of the user information notification and the user information from the camera device. When the user area corresponding to the user is included in the first image data, second image data is generated by performing privacy protection processing for protecting the privacy of the user with respect to the user area, and the generated second image data is stored. A process of notifying the user that the second image data has been stored. A program for causing the above to be executed.

Images