Information processing system, information processing method, and information processing device

The system allows authorized access to unmasked images and unauthorized access to masked images, addressing the challenge of managing personal information dissemination on SNS platforms.

JP7585788B2Active Publication Date: 2024-11-19SONY GROUP CORP
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
JP2020541092
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2018-09-06
Filing Date
2019-08-09
Publication Date
2024-11-19
Estimated Expiration
2039-08-09

AI Technical Summary

Technical Problem

Conventional privacy protection measures for images on social networking services (SNS) prevent viewing of the original, unprocessed images, making it difficult to manage and control the dissemination of personal information.

Method used

An information processing system and method that allows viewing both the original and processed images, using a server and terminal device with image detection and processing units to apply mask processing based on access permissions, ensuring authorized access can view unmasked images while unauthorized access sees masked images.

Benefits of technology

Enables secure management of personal information by allowing authorized users to view unmasked images while preventing unauthorized access, thereby controlling the spread of personal information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007585788000001
    Figure 0007585788000001
  • Figure 0007585788000002
    Figure 0007585788000002
  • Figure 0007585788000003
    Figure 0007585788000003
Patent Text Reader

Abstract

The terminal device (30) includes a solid-state imaging element (3003) including an image sensor (3100) that converts received light into an image, an image processing unit (3104) that detects a specific object in the image, and an output interface (3103) that outputs the image, and a first communication unit (303) that transmits the image output from the output interface. The information processing device (10) includes a second communication unit (100) and a processor (1000), and the processor receives an image transmitted from the terminal device by the second communication unit, and performs masking processing to mask the specific object when the image received by the second communication unit is an image in which the specific object has been detected by the image processing unit.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to an information processing system, an information processing method, Law and an information processing device. [Background technology]

[0002] With the development of the Internet, services such as SNS (Social Networking Service) that are available only to users who register an account and use specific application software are becoming widespread. In addition, there are also increasing opportunities for users to publish images that they have taken on such services. In general, in a service such as SNS, access is limited to the service, so that even if a captured image published on the service contains personal information such as a face, it is possible to prevent the image from developing into a major problem.

[0003] On the other hand, when a captured image published on an SNS is taken outside the SNS, the captured image may be leaked onto the Internet. Since it is extremely difficult to recover information once leaked onto the Internet, it is necessary for the person publishing the captured image or the SNS to take some measures to protect privacy in advance. As an example of such measures, a technology is known that, when a face is detected from a captured image, performs some processing such as masking processing on the detected face area to make it difficult to identify the face, thereby protecting privacy (for example, Patent Document 1). By performing such processing on the captured image to be published, privacy can be protected even if the captured image is taken outside the SNS. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2009-194687 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in conventional privacy protection measures for captured images, captured images are directly processed, so that the original, unprocessed captured images cannot be viewed even within an SNS.

[0006] The present disclosure relates to an information processing system and an information processing method that enable both viewing of an original image before processing and viewing of an image after processing. Law The present invention aims to provide an information processing device. [Means for solving the problem]

[0007] To achieve the above objectives The information processing system according to the present disclosure , converting the received light into an image and output as a captured image and an image sensor for detecting the Imaging For images Includes open windows An image processing unit that detects a specific object; Imaging a solid-state imaging device including an output interface for outputting an image; The imaging A terminal device including a first communication unit for transmitting an image, and an information processing device including a second communication unit and a processor, Information processing device teeth, According to instructions of said processor, The terminal device transmits The imaging The image is transmitted by the second communication unit. As received image and received by the second communication unit. Reception The image is an image in which the specific object is detected by the image processing unit, and is received by the second communication unit. Reception When an access request for an image is received from a source other than the terminal device, the source is Reception Whether you are allowed to access the image or not The processor Judging, The processor detects the source. Applicable Reception If it is determined that access to the image is not permitted, Reception image AgainstMasking by replacing the specific object with another image and applying the mask processing to the received image. Sending the image to the source; The processor detects the source. Applicable Reception If it is determined that access to the image is permitted, Reception image The above mask Without processing Send it to the sender. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an information processing system that can be applied to each embodiment. [Diagram 2] 2 is a diagram illustrating an outline of a processing flow by the information processing system according to the first embodiment. FIG. [Diagram 3] FIG. 2 is a block diagram showing an example of a hardware configuration of a server applicable to the first embodiment. [Figure 4] 2 is a block diagram showing an example of a hardware configuration of a terminal device applicable to the first embodiment. FIG. [Diagram 5] 2 is a block diagram showing an example of a hardware configuration of an imaging unit applicable to the first embodiment. FIG. [Figure 6] 1 is a diagram showing an example in which an imaging section according to the first embodiment is formed using a stacked CIS having a two-layer structure. FIG. [Figure 7] 1 is a diagram showing an example in which an imaging section according to the first embodiment is formed using a stacked CIS having a three-layer structure. FIG. [Figure 8] FIG. 2 is a functional block diagram illustrating an example of functions of a terminal device according to the first embodiment. [Figure 9] FIG. 2 is a functional block diagram illustrating an example of a function of a server according to the first embodiment. [Figure 10] 10 is a flowchart illustrating an example of a captured image upload process performed by the terminal device according to the first embodiment. [Figure 11] 4A to 4C are diagrams each showing a schematic diagram of a state in which a specific object is detected from a captured image, which is applicable to the first embodiment. [Figure 12]4A to 4C are diagrams each showing a schematic diagram of a state in which a specific object is detected from a captured image, which is applicable to the first embodiment. [Figure 13] 6 is a flowchart illustrating an example of a process in a server according to the first embodiment. [Figure 14] 5A to 5C are diagrams showing an example in which mask processing is performed on rectangular areas included in a captured image according to the first embodiment. [Figure 15] 5A to 5C are diagrams showing an example in which mask processing is performed on rectangular areas included in a captured image according to the first embodiment. [Figure 16] FIG. 2 is a diagram for roughly explaining an example of a method for encrypting a rectangular area in an image compressed and encoded in the JPEG format, which is applicable to the first embodiment. [Figure 17] 4 is a sequence chart illustrating an example of an overall operation of the information processing system according to the first embodiment. [Figure 18] 10 is a flowchart illustrating an example of a process in a server according to the second embodiment. [Figure 19] 10 is a sequence chart illustrating an example of an overall operation of an information processing system according to a second embodiment. [Figure 20] 11 is a diagram illustrating an example of a configuration of an information processing system according to a third embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0010] [System overview applicable to each embodiment] Fig. 1 is a diagram illustrating an example of a configuration of an information processing system applicable to each embodiment. In Fig. 1, the information processing system 1 includes, as a hardware configuration, a server 10 and a terminal device 30 connected by a network 2, which is, for example, the Internet.

[0011] The server 10 is composed of, for example, a plurality of information processing devices and a storage device in a distributed manner on the cloud network 3. Without being limited to this, the server 10 may be composed of, for example, one information processing device including a storage device. The terminal device 30 is, for example, a mobile terminal such as a smartphone or a tablet computer, and includes an imaging function. A user 31 can transmit an image captured using the imaging function of the terminal device 30 to the network 2.

[0012] The information processing system 1 may further include terminal devices 40 and 41 connectable to the network 2. In this example, the terminal device 40 is shown as a mobile terminal, and the terminal device 41 is shown as a desktop personal computer (PC).

[0013] In the information processing system 1, the server 10 provides a closed service, such as a social networking service (SNS), to a person who has registered an account. The terminal devices 30, 40, and 41 are equipped with specific application software, and the terminal devices 30, 40, and 41 can use the service by registering an account on the server 10 using the application software. The application software is installed in each of the terminal devices 30, 40, and 41 and used.

[0014] Without being limited to this, the application software may be one that runs on the server 10. In this case, the service can be used by accessing the server 10 from a browser application installed on each of the terminal devices 30, 40, and 41. In the following description, it is assumed that the application software is installed on the terminal devices 30 and 40. In the following description, it is assumed that the service provided by the server 10 is an SNS, and the application software for using the service is called an SNS app.

[0015] A user 31 of a terminal device 30 sets account information including identification information (e.g., a username and a password) for identifying the user 31 using an SNS app installed in the terminal device 30, and transmits the set account information to the server 10 to register the account. The server 10 authenticates the user 31 based on the account information transmitted from the terminal device 30, and if the authentication is successful, permits the user 31 to log in to the SNS.

[0016] A user 31 can log in to an SNS using an SNS app on the terminal device 30, and transmit an image captured using, for example, an imaging function of the terminal device 30 to the server 10 via the network 2. The server 10 can store the captured image transmitted from the terminal device 30 and can also make the captured image publicly available through the SNS.

[0017] Similarly, it is assumed that the user (not shown) of the terminal device 40 has registered account information in the server 10 using the SNS application.

[0018] [First embodiment] (Overview of Processing According to the First Embodiment) Next, an information processing system according to the first embodiment will be described. Fig. 2 is a diagram that shows a schematic flow of processing by the information processing system 1 according to the first embodiment. A user 31 accesses the server 10 by an SNS application on a terminal device 30 and logs into the SNS. After logging into the SNS, the user 31 captures an image of a subject using an imaging function of the terminal device 30 (step S1). In the terminal device 30, the SNS application analyzes the captured image and detects a specific object included in the captured image (step S2).

[0019] Here, the specific object to be detected by the terminal device 30 refers to an image included in the captured image that shows personal information itself or may contain personal information. More specifically, an image of a face is an image that shows personal information itself. An image of a vehicle license plate is also considered to be an image that shows personal information itself. An image of an open window of a building may contain personal information. When a specific object is detected from a captured image in the terminal device 30, the SNS application adds identification information (called a detection flag) indicating this to the captured image.

[0020] The terminal device 30 transmits the captured image to the server 10 via the network 2 using the SNS application (step S3). If a detection flag indicating that a specific object has been detected is added to the captured image received from the terminal device 30, the server 10 analyzes the captured image to detect the specific object contained in the captured image. The server 10 then performs reversible masking on the detected specific object (step S4). The masking process renders the captured image in a state in which the original image of the specific object cannot be distinguished. The server 10 stores the captured image that has been subjected to the masking process (referred to as a masked image) in a storage device such as a hard disk drive, and registers it on the SNS.

[0021] The process of acquiring a masked image registered in an SNS in the server 10 will be described below. First, an example will be described in which a user who has an account registered in an SNS acquires a masked image registered in the SNS using a terminal device 40 equipped with an SNS app. The user accesses the server 10 using the SNS app on the terminal device 40 to log in to the SNS and request the masked image from the server 10 (step S10). This request for a masked image by the terminal device 40 is made through legitimate access by logging in to the SNS.

[0022] A request for a masked image may be regarded as a user logging in to an SNS from the terminal device 30, or as a user switching a page of a display screen by an SNS app while operating the SNS app. Also, a request for a masked image may be regarded as a user scrolling a page of a display screen by the SNS app or enlarging or reducing the display on the display screen while operating the SNS app.

[0023] In addition, the user does not need to be aware of whether or not the image displayed on the display screen of the SNS app has been masked, and requesting a masked image here includes a process in which the user operates the SNS app without being aware of the presence of image information.

[0024] In response to the request for the masked image made through this legitimate access, the server 10 performs a removal process to remove the mask applied to the masked image (step S11). This allows the terminal device 40 to obtain an image in which the mask of the masked image has been removed (referred to as an unmasked image) through the SNS application (step S12).

[0025] Next, an example will be described in which a masked image registered in an SNS is acquired from, for example, the terminal device 41 without accessing the SNS. The terminal device 41 accesses the server 10 based on the address of the masked image acquired by some method, and requests the masked image without logging in to the SNS (step S20). This request for a masked image by the terminal device 41 is made through unauthorized access without logging in to the SNS.

[0026] The server 10 does not perform any processing related to the mask in response to the request for the masked image made through this unauthorized access. Therefore, the terminal device 41 acquires the masked image in its original state, and is unable to recognize the original image at the masked position.

[0027] In this way, in the first embodiment, the server 10 releases the masking of the masked image to generate an unmasked image when a request for a masked image registered in an SNS is made through legitimate access by logging in to the SNS, whereas the server 10 does not release the masking of the masked image when a request for a masked image is made through unauthorized access without logging in to the SNS.

[0028] Therefore, in the information processing system 1 according to the first embodiment, it is possible to simultaneously view an image with the masking removed from an image containing a specific object registered on an SNS and a masked image. At this time, the information processing system 1 removes the processing, i.e., the masking, for an image requested by a user under the management of the SNS, and leaves the masking applied for an image requested by a user outside the management of the SNS. Therefore, it is possible to suppress the spread of unmasked images, which are obtained by removing the masking from a masked image registered on an SNS, for example, on the Internet.

[0029] (More detailed configuration of the information processing system according to the first embodiment) Next, a more detailed configuration of the information processing system 1 according to the first embodiment will be described.

[0030] Fig. 3 is a block diagram showing an example of a hardware configuration of a server 10 applicable to the first embodiment. Note that, in this description, the server 10 is assumed to be configured by one information processing device. In Fig. 3, the server 10 includes a CPU (Central Processing Unit) 1000, a ROM (Read Only Memory) 1001, a RAM (Random Access Memory) 1002, a storage 1003, and a communication I / F 1004, each of which is connected to a bus 1010.

[0031] The storage 1003 is a non-volatile large-capacity storage medium such as a hard disk drive, a flash memory, etc. The storage 1003 may be built into the hardware of the server 10, or may be an external storage device for the server 10.

[0032] The CPU 1000 uses the RAM 1002 as a working memory in accordance with programs stored in the ROM 1001 and the storage 1003 to control the overall operation of the server 10. The programs include a program for providing the above-mentioned SNS. The communication I / F 1004 controls communication with the network 2 in accordance with instructions from the CPU 1000.

[0033] Fig. 4 is a block diagram showing an example of a hardware configuration of a terminal device 30 applicable to the first embodiment. In Fig. 4, the terminal device 30 includes a CPU 3000, a ROM 3001, a RAM 3002, an imaging unit 3003, a display control unit 3004, a storage 3005, an input device 3006, a data I / F 3007, and a communication I / F 3008, each of which is connected to a bus 3010.

[0034] The storage 3005 is a non-volatile large-capacity storage medium such as a hard disk drive or a flash memory. The CPU 3000 uses the RAM 3002 as a working memory and controls the overall operation of the terminal device 30 in accordance with the programs stored in the ROM 3001 and the storage 3005. The programs include the above-mentioned SNS application.

[0035] The imaging unit 3003 includes an imaging element, an optical system (such as a lens) for irradiating the imaging element with light from a subject, and a configuration for controlling the imaging element and processing signals output from the imaging element. The imaging unit 3003 captures still images or moving images under the control of the CPU 3000. The captured images captured by the imaging unit 3003 are stored in the RAM 3002. In the following, unless otherwise specified, the captured images captured by the imaging unit 3003 are considered to be still images.

[0036] The display control unit 3004 generates a display signal that can be displayed by the display device 3020 based on a display control signal generated by the CPU 3000. The display device 3020 includes, for example, an LCD (Liquid Crystal Display) and a drive circuit for driving the LCD. The input device 3006 can be, for example, a pointing device that outputs coordinate information according to a position touched by a finger or the like. It is preferable to configure the input device 3006 and the display device 3020 as a single unit, that is, as a so-called touch panel.

[0037] The data I / F 3007 is an interface for transferring data to and from an external device, and may be implemented using, for example, a Universal Serial Bus (USB) or Bluetooth (registered trademark). The communication I / F 3008 controls communication with the network 20 in accordance with instructions from the CPU 3000.

[0038] Fig. 5 is a block diagram showing an example of a hardware configuration of an imaging unit 3003 applicable to the first embodiment. Note that an optical system is omitted in Fig. 5. The imaging unit 3003 includes a pixel unit 3100, a signal processing unit 3101, a memory unit 3102, an I / F unit 3103, an image processing unit 3104, and an element control unit 3105, and these units are a CMOS image sensor (CIS) integrally formed using a complementary metal oxide semiconductor (CMOS).

[0039] The pixel unit 3100 is provided with a plurality of scanning lines and a plurality of signal lines, and a pixel circuit is arranged at each intersection of each scanning line and each signal line, so that a plurality of pixel circuits are arranged in a two-dimensional lattice. For example, the pixel circuit uses a photodiode as a photoelectric conversion element, and a microlens and a color filter are formed on the photodiode. The pixel unit 3100 also includes a row scanning circuit, an AD (Analog to Digital) converter for each row, and a column scanning circuit. In the pixel unit 3100, each pixel circuit is scanned row by row by the row scanning circuit, and the analog pixel signal output for each row is converted into digital pixel data by the AD converter. The pixel data converted into AD for each row is transferred to the signal processing unit 3101 in sequence by the column scanning circuit.

[0040] The signal processing unit 3101 performs signal processing such as CDS (Correlated Double Sampling) processing, AGC (Auto Gain Control) processing, and WB (White Balance) processing on the pixel data transferred from the pixel unit 3100. The image data subjected to the signal processing by the signal processing unit 3101 is transferred to the memory unit 3102 and the I / F unit 3103. The memory unit 3102 stores the image data subjected to the signal processing by the signal processing unit 3101. The memory unit 3102 has a capacity capable of storing, for example, one frame's worth of image data (all pixel circuits of the pixel unit 3100).

[0041] The image processing unit 3104 is, for example, a DSP (Digital Signal Processor), and performs a predetermined image processing on the image data stored in the memory unit 3102. In the first embodiment, the image processing unit 3104 performs a detection process to detect a predetermined specific object included in an image (captured image) based on the image data stored in the memory unit 3102. The image processing unit 3104 detects the specific object using, for example, a detection process based on pattern recognition using a known technology. Without being limited to this, the image processing unit 3104 may detect the specific object using a method such as deep learning. When a specific object is detected from the captured image stored in the memory unit 3102, the image processing unit 3104 writes a detection flag indicating that fact into the memory unit 3102.

[0042] The I / F unit 3103 is an interface that controls communication between the imaging unit 3003 and the outside. For example, an interface compatible with MIPI (Mobile Industry Processor Interface) can be applied to the I / F unit 3103. However, the I / F unit 3103 may apply an interface of a unique system to the imaging unit 3003.

[0043] For example, the I / F unit 3103 outputs image data transferred from the signal processing unit 3101 or image data (captured image) stored in the memory unit 3102 to the outside of the imaging unit 3003. The I / F unit 3103 outputs designated image data from the image data transferred from the signal processing unit 3101 and the image data stored in the memory unit 3102 to the outside, for example, in accordance with an instruction from the outside.

[0044] The element control unit 3105 controls the overall operation of the imaging unit 3003. For example, the element control unit 3105 generates a timing signal for synchronously controlling the operations of the row scanning circuit, the AD converter, and the column scanning circuit in the pixel unit 3100. The element control unit 3105 also controls the timing for the image processing unit 3104 to start image processing.

[0045] In the above description, the pixel unit 3100, the signal processing unit 3101, the memory unit 3102, the I / F unit 3103, the image processing unit 3104, and the element control unit 3105 included in the imaging unit 3003 are formed on one substrate, but this is not limited to the example. For example, the imaging unit 3003 may be a stacked CIS formed by stacking multiple semiconductor chips.

[0046] As an example, the imaging unit 3003 can be formed by a two-layer structure in which semiconductor chips are stacked in two layers. FIG. 6 is a diagram showing an example in which the imaging unit 3003 according to the first embodiment is formed by a two-layer stacked CIS. In the structure of FIG. 6, a pixel unit 3100 is formed in a semiconductor chip of a first layer, and a memory + logic unit 3200 is formed in a semiconductor chip of a second layer. The memory + logic unit 3200 includes, for example, a signal processing unit 3101, a memory unit 3102, an I / F unit 3103, an image processing unit 3104, and an element control unit 3105. Note that the semiconductor chip of the first layer may be configured to include the pixel unit 3100 and the signal processing unit 3101. As shown on the right side of FIG. 6, the semiconductor chip of the first layer and the semiconductor chip of the second layer are bonded together while being in electrical contact with each other, thereby forming the imaging unit 3003 as one solid-state imaging element.

[0047] As another example, the imaging unit 3003 can be formed by a three-layer structure in which semiconductor chips are stacked in three layers. FIG. 7 is a diagram showing an example in which the imaging unit 3003 according to the first embodiment is formed by a stacked CIS with a three-layer structure. In the structure of FIG. 7, the pixel unit 3100 is formed in the semiconductor chip of the first layer, the memory unit 3102 is formed in the semiconductor chip of the second layer, and the logic unit 3201 is formed in the semiconductor chip of the third layer. In this case, the logic unit 3201 includes, for example, a signal processing unit 3101, an I / F unit 3103, an image processing unit 3104, and an element control unit 3105. Note that the semiconductor chip of the first layer may be configured to include the pixel unit 3100 and the signal processing unit 3101. As shown on the right side of FIG. 7, the semiconductor chip of the first layer, the semiconductor chip of the second layer, and the semiconductor chip of the third layer are bonded together while being in electrical contact with each other, thereby forming the imaging unit 3003 as one solid-state imaging element.

[0048] Fig. 8 is a functional block diagram of an example for explaining functions of the terminal device 30 according to the first embodiment. In Fig. 8, the terminal device 30 includes an image capture processing unit 300, an object detection unit 301, an application unit 302, and a communication unit 303. The application unit 302 includes an encoding unit 320, a UI unit 321, and an authentication request unit 322.

[0049] Of these units, the image capture processing unit 300 and the object detection unit 301 are functions realized in the image capture unit 3003 described with reference to Fig. 5. For example, the image capture processing unit 300 corresponds to the functions of the pixel unit 3100, the signal processing unit 3101, and the element control unit 3105 in Fig. 5. Moreover, the object detection unit 301 corresponds to the functions of the memory unit 3102 and the image processing unit 3104 in Fig. 5.

[0050] In FIG. 8, the function of the application unit 302 is realized by, for example, the above-mentioned SNS application.

[0051] A program (SNS application) for realizing each function of the application unit 302 (encode unit 320, UI unit 321, and authentication request unit 322) in the terminal device 30 is stored on a computer connected to a network such as the Internet and is provided by being downloaded via the network. The program may also be configured to be provided or distributed via a network such as the Internet. Furthermore, the program may be provided by recording it in an installable or executable file format on a computer-readable recording medium such as a CD (Compact Disk), a flexible disk (FD), or a DVD (Digital Versatile Disk).

[0052] The program has a modular configuration including an encoding unit 320, a UI unit 321, and an authentication request unit 322. In terms of actual hardware, the CPU 3000 reads out the program from a storage medium such as ROM 3001 or storage 3005 and executes it, whereby the above-mentioned units are loaded onto a main storage device such as RAM 3002, and the encoding unit 320, the UI unit 321, and the authentication request unit 322 are generated on the main storage device.

[0053] In the application section 302, the encoding section 320 performs encoding processing such as compression encoding on the captured image output from the imaging section 3003. Here, it is assumed that the JPEG (Joint Photographic Experts Group) method is applied as the compression encoding method applied to the captured image. In the JPEG method, an image is divided into blocks of a fixed size (e.g., 8 pixels x 8 pixels), and a discrete cosine transform is performed on each block to convert image information from the spatial domain to information in the frequency domain. Hereinafter, this block is referred to as a DCT (Discrete Cosine Transform) block. The encoding section 320 performs a quantization process and a Huffman compression process on the image converted into information in the frequency domain to reduce the amount of information, and compression-encodes the original image.

[0054] The encoding unit 320 stores the compressed and encoded image in a file in a predetermined format. At this time, the encoding unit 320 can further store metadata related to the image in the file. When compressing and encoding an image in the JPEG format, the encoding unit 320 can store the compressed and encoded image in a file in the Exif (Exchangeable image file format) format. The metadata is stored in an application marker segment area defined in the Exif format.

[0055] The UI (User Interface) unit 321 provides a user interface for user operations. The UI unit 321 generates a display control signal for displaying an operation screen on the display device 3020. The UI unit 321 also accepts a user operation on the input device 3006, and controls the operation of each unit (e.g., the imaging unit 3003 and the authentication request unit 322) of the terminal device 30 in response to the user operation.

[0056] The authentication request unit 322 transmits authentication information (account information) for logging in to the SNS and an authentication request for the authentication information to the server 10, and requests authentication of the terminal device 30. The authentication request unit 322 can store, for example, the account information input by the user 31 in a non-volatile manner in a predetermined area of ​​the storage 3005. The authentication request unit 322 can automatically transmit the stored account information to the server 10 when the SNS application is started.

[0057] The communication unit 303 performs communication with the network 2 under the control of the UI unit 321. The captured image encoded by the encoding unit 320 and stored in a file, and the account information transmitted by the authentication request unit 322 are transmitted by the communication unit 303.

[0058] Fig. 9 is a functional block diagram of an example for explaining the function of the server 10 according to the first embodiment. In Fig. 9, the server 10 includes a communication unit 100, an authentication unit 101, a decoding unit 102, an analysis unit 103, a mask processing unit 104, and a storage unit 105. The communication unit 100, the authentication unit 101, the decoding unit 102, the analysis unit 103, the mask processing unit 104, and the storage unit 105 are realized by a program that runs on the CPU 1000. However, the communication unit 100, the authentication unit 101, the decoding unit 102, the analysis unit 103, the mask processing unit 104, and the storage unit 105 may be partly or entirely configured by hardware circuits that operate in cooperation with each other.

[0059] The communication unit 100 communicates with the network 2. The authentication unit 101 performs authentication processing in response to an authentication request including account information received by the communication unit 100. The decoding unit 102 decodes the image received by the communication unit 100 to decode the compressed code, and acquires metadata added to the image. The analysis unit 103 analyzes the decoded image decoded by the decoding unit 102, and detects a specific object included in the decoded image.

[0060] The mask processing unit 104 performs mask processing on the image according to the analysis result of the analysis unit 103. At this time, the mask processing unit 104 can perform mask processing on the compressed and encoded image before it is decoded by the decoding unit 102. The mask processing unit 104 also removes the mask processing performed on the image. The mask processing unit 104 executes mask processing or mask removal processing according to the authentication result by the authentication unit 101.

[0061] The memory unit 105 stores the image output from the mask processing unit 104, for example, in the storage 1003. The memory unit 105 also performs processing to read an image from the storage 1003. Hereinafter, unless otherwise specified, processing by the memory unit 105 to store an image in the storage 1003 will be described as "storing an image in the memory unit 105," and processing to read an image from the storage 1003 will be described as "reading an image from the memory unit 105."

[0062] The program for realizing each function according to the first embodiment in the server 10 is provided by recording it in an installable or executable format on a computer-readable recording medium such as a CD (Compact Disk), a flexible disk (FD), or a DVD (Digital Versatile Disk). Without being limited to this, the program may be provided by storing it on a computer connected to a network such as the Internet and downloading it via the network. The program may also be configured to be provided or distributed via a network such as the Internet.

[0063] The program has a modular configuration including a communication unit 100, an authentication unit 101, a decoding unit 102, an analysis unit 103, a mask processing unit 104, and a storage unit 105. In terms of actual hardware, the CPU 1000 reads out the program from a storage medium such as a ROM 1001 or a storage 1003 and executes it, whereby the above-mentioned units are loaded onto a main storage device such as a RAM 1002, and the communication unit 100, the authentication unit 101, the decoding unit 102, the analysis unit 103, the mask processing unit 104, and the storage unit 105 are generated on the main storage device.

[0064] (Details of Processing According to the First Embodiment) Next, the processing in the information processing system 1 according to the first embodiment will be described in more detail. Fig. 10 is a flowchart of an example showing a captured image upload process by the terminal device 30 according to the first embodiment. It is assumed that, prior to execution of the process according to the flow of Fig. 10, the terminal device 30 is equipped with application software (SNS application) corresponding to a service (e.g., SNS) provided by the information processing system 1 according to the first embodiment. It is also assumed that the user 31 of the terminal device 30 has registered account information for logging in to the SNS in advance in the server 10.

[0065] In step S100, an SNS application is launched in response to a user operation on the terminal device 30. When the SNS application is launched, the authentication request unit 322 transmits account information to the server 10 and requests authentication. The server 10 performs authentication using the authentication unit 101 based on the account information transmitted from the terminal device 30, and if the authentication is successful, the SNS provided by the server 10 becomes available to the terminal device 30.

[0066] The terminal device 30 captures an image in response to a user operation to instruct imaging by the imaging processing unit 300 (step S101). The captured image obtained by imaging is passed to the object detection unit 301. In the next step S102, the terminal device 30 executes a detection process of a specific object on the captured image passed from the imaging processing unit 300 by the object detection unit 301. The specific object is, for example, an image that can identify personal information, and may be a face, a vehicle license plate, an open window, or the like. The object detection unit 301 detects the specific object using a detection method based on known pattern recognition. The detection process of the specific object may be executed by the object detection unit 301 and the CPU 3000 of the terminal device 30 in cooperation with each other.

[0067] Note that, here, the captured image captured and acquired in step S101 is applied as a target for specific object detection in step S102, but this is not limited to this example. For example, the terminal device 30 may perform specific object detection by the object detection unit 301 in step S102 on a captured image captured by the imaging unit 3003 and then stored in the storage 3005 or the like. For example, the object detection unit 301 executes a detection process of a specific object on a captured image read from the storage 1003 in response to a user operation or the like.

[0068] In the next step S103, the terminal device 30 determines whether or not a specific object has been detected from the captured image by the object detection unit 301. When the object detection unit 301 determines that at least one specific object has been detected from the captured image (step S103, "Yes"), the process proceeds to step S104.

[0069] In step S104, the terminal device 30 adds a detection flag indicating that a specific object has been detected by the object detection unit 301 to the captured image. The detection flag, for example, has a value of "1" indicating that a specific object has been detected, and a value of "0" indicating that a specific object has not been detected. The detection flag has a default value of "0". In step S104, the value of the detection flag is changed from "0" to "1". As described above, when the object detection process is executed by the object detection unit 301 and the CPU 3000 in cooperation with each other, the detection flag may be added by the object detection unit 301 or by the CPU 3000.

[0070] 11 and 12 are diagrams that are applicable to the first embodiment and are schematic diagrams showing a state in which a specific object is detected from a captured image. FIG. 11 shows an example in which the specific object is a face. In the example of FIG. 11, in a captured image 50a, each rectangular region 51a including a face is detected as a specific object region. FIG. 12 shows an example in which the specific object is an open window. In the example of FIG. 12, similar to the example of FIG. 11, in a captured image 50b, each rectangular region 51b including an open window is detected as a specific object region.

[0071] On the other hand, if it is determined in step S103 that no specific object is detected from the captured image (step S103, “No”), the object detection unit 301 skips the process of step S104 and moves the process to step S105. In this case, the value of the detection flag is maintained at “0”.

[0072] In step S105, the terminal device 30 causes the encoding unit 320 to perform encoding processing on the captured image acquired in step S101. For example, the encoding unit 320 compresses and encodes the captured image using the JPEG format, and stores the compressed and encoded image in a file in Exif format. The encoding unit 320 also includes a detection flag in metadata of the captured image, and stores the metadata in an application marker segment area defined in the Exif format.

[0073] In the next step S106, the terminal device 30 transmits, via the communication unit 303 in response to a user operation, the captured image that was encoded and stored in the file in step S105 to the server 10.

[0074] In the above description, the process according to the flowchart in Fig. 10 has been described as performing specific object detection on still images, but this is not limited to this example. For example, the process according to the flowchart in Fig. 10 may be performed on moving images. In this case, in step S102, the terminal device 30 executes a specific object detection process for each frame of the moving images using the object detection unit 301. In the terminal device 30, the object detection unit 301 adds a detection flag with a value of "1" to a frame in which a specific object has been detected (step S103, "Yes") as metadata for the frame (step S104).

[0075] In step S105, the terminal device 30 uses the encoding unit 320 to compress and encode the video using a predetermined method such as the H.264 (ITU-T Rec. H.264 | ISO / IEC 14496-10 Advanced Video Coding) method, and transmits the compressed and encoded video to the server 10 in step S106. In the above description, the detection flag is added to each frame as metadata, but this is not limited to this example. For example, data in which detection flags for multiple frames are associated with frame numbers may be created separately and added to the video data as metadata. Furthermore, the terminal device 30 may transmit the video to the server 10 as a file or as stream data.

[0076] 13 is a flowchart of an example showing processing in the server 10 according to the first embodiment. In step S200, the server 10 detects access via the network 2 using the communication unit 100. In the next step S201, the server 10 determines using the authentication unit 101 whether the access detected in step S200 is access from an access source that has already received an authentication request.

[0077] When the authentication unit 101 determines that the access detected in step S200 is not an access for which an authentication request has already been received (step S201, "No"), the server 10 ends the series of processes according to the flowchart in Fig. 13. Alternatively, the authentication unit 101 may request account information for the access to prompt the authentication process.

[0078] On the other hand, if the authentication unit 101 determines that the access detected in step S200 is an access for which an authentication request has already been received (step S201, "Yes"), the server 10 shifts the process to step S202. In step S202, the authentication unit 101 determines whether or not authentication has been successful for this received authentication request. If the authentication unit 101 determines that authentication has not been successful for the received authentication request (step S202, "NG"), the authentication unit 101 ends the series of processes according to the flowchart of FIG. 13.

[0079] On the other hand, if the authentication unit 101 determines that the authentication for the received authentication request has been successful (step S202, "OK"), the server 10 shifts the process to step S203. In step S203, the server 10 determines, for example, by the communication unit 100, whether the access detected in step S200 is a request for image registration or acquisition. If the communication unit 100 determines that the access detected in step S200 is a request for image registration (step S203, "Register"), the server 10 shifts the process to step S204. On the other hand, if the communication unit 100 determines that the access detected in step S200 is a request for image acquisition (step S203, "Acquire"), the server 10 shifts the process to step S210.

[0080] In addition, if the communication unit 100 determines in step S203 that the access detected in step S200 is neither image registration nor image acquisition, it can terminate the series of processes according to the flowchart of Figure 13, for example, and transition to other processes.

[0081] In step S204, the server 10 receives the file storing the image transmitted from the access source detected in step S200, via the communication unit 100. This access source is, for example, the terminal device 30, and the communication unit 100 receives the file storing the captured image transmitted in step S106 in the flowchart of FIG.

[0082] In the next step S205, the server 10 uses the decoding unit 102 to extract and decode the compressed and encoded captured image from the file received in step S204, thereby decoding the captured image. The decoded captured image is bitmap data having luminance information and color information for each pixel. The decoding unit 102 also holds the compressed and encoded captured image stored in the file. The decoding unit 102 also obtains metadata from the file.

[0083] In the next step S206, the server 10 determines whether or not the metadata acquired in step S205 includes a detection flag indicating that a specific object has been detected, using the decoding unit 102. If the decoding unit 102 determines that the detection flag is not included (step S206, "No"), the process proceeds to step S209. On the other hand, if the decoding unit 102 determines that the detection flag is included (step S206, "Yes"), the process proceeds to step S207.

[0084] In step S207, the server 10 analyzes the captured image decoded in step S205 by the analysis unit 103, detects all detectable specific objects from the captured image, and acquires rectangular regions including each of the detected specific objects. The analysis unit 103 passes the coordinates of the acquired rectangular regions including all of the specific objects (hereinafter, referred to as specific object regions) to the mask processing unit 104.

[0085] In the next step S208, the server 10 performs reversible mask processing using the mask processing unit 104 on the area corresponding to the specific object area of ​​the compressed and encoded captured image held by the decoding unit 102 in step S204, based on the coordinates of the specific object area passed from the analysis unit 103.

[0086] FIG. 14 is a diagram showing an example in which masking is performed on each rectangular area 51a included in the captured image 50a shown in FIG. 11 according to the first embodiment. In FIG. 14, the captured image 50a' has a mask applied to each rectangular area 52a corresponding to each rectangular area 51a in FIG. 11, and the faces included in each rectangular area 51a are indistinguishable. Similarly, FIG. 15 is a diagram showing an example in which masking is performed on each rectangular area 51b included in the captured image 50b shown in FIG. 12 according to the first embodiment. In FIG. 15, the captured image 50b' has a mask applied to each rectangular area 52b corresponding to each rectangular area 51b in FIG. 12, and the open windows included in each rectangular area 51b are indistinguishable.

[0087] In this manner, the masking process can make objects that are included in the original captured images 50a and 50b and that can identify personal information unidentifiable.

[0088] The reversible mask processing by the mask processing unit 104 will be described using a more specific example. The mask processing unit 104 rearranges information within a rectangular area in an image, making the information within the rectangular area difficult to distinguish, and can mask the rectangular area. In addition, by rearranging the information within the masked rectangular area back to its original order, the mask can be removed and the information within the rectangular area can be restored to its original, easily distinguishable form.

[0089] The mask processing unit 104 rearranges the information in the rectangular area, for example, by encrypting the information in the rectangular area. The mask processing unit 104 encrypts and masks the information in the rectangular area using a pre-given key, decrypts the encryption in the rectangular area using the key, and rearranges the information in the rectangular area to its original order. As an example of such encryption, a method of rearranging information by performing substitution and transposition according to a predetermined key can be applied. Without being limited to this, encryption using a hash function such as SHA-1 (Secure Hash Algorithm-1) or MD5 (Message Digest Algorithm 5) may also be used.

[0090] Here, when an image is compressed and encoded in the JPEG format, it is necessary to encrypt the data format of this JPEG format. An example of a method for encrypting a rectangular area in an image compressed and encoded in the JPEG format, which is applicable to the first embodiment, will be outlined with reference to Fig. 16. In Fig. 16, an image 50c is a bitmap image, and includes a face 501 as a specific object. Generally, in the JPEG format, as described above, compression encoding is performed in units of DCT blocks having a fixed size of 8 pixels x 8 pixels.

[0091] For example, an image 50c is divided into 8 pixel by 8 pixel blocks 500 corresponding to the DCT blocks, and a rectangular area 502 consisting of a plurality of blocks 500 including a face 501 is extracted. In data compressed and encoded by the JPEG method, encryption may be performed on a DCT block basis corresponding to each block 500 included in this rectangular area 502.

[0092] In the above description, the mask processing unit 104 performs the mask processing for each specific object region, but this is not limited to this example. For example, the mask processing unit 104 may encrypt the entire captured image included in the file received in step S204. Furthermore, the mask processing unit 104 can also encrypt the file itself received in step S204.

[0093] In the above description, the mask processing unit 104 performs mask processing on the compressed and encoded captured image, but this is not limited to this example. For example, the mask processing unit 104 may perform mask processing on the captured image that has been decoded in step S205 and converted into a bitmap image, and then compress and encode the captured image based on the masked bitmap image again using a JPEG format or the like.

[0094] Returning to the explanation of the flowchart in FIG. 13, when the server 10 performs the masking process by the masking processor 104 in step S208, the server 10 shifts the process to step S209.

[0095] In step S209, the server 10 stores the captured image (image file) that has been subjected to the mask process in step S208 by the storage unit 105, or the captured image (image file) that has been received in step S204 when the process has shifted from step S206 to this step S209, in the storage 1003, for example, and registers the captured image in the SNS. At this time, the storage unit 105 stores information indicating the positions of the specific object regions detected in step S207 in the storage 1003 in association with the captured image. When the captured image is registered in the SNS, the series of processes according to the flowchart in FIG. 13 is terminated.

[0096] If the image received in step S204 is a compressed and encoded video, the server 10 causes the decoding unit 102 to decode the compressed code of the video received in step S205 to generate a bitmap image for each frame (step S205). The server 10 also causes the analysis unit 103 to detect a specific object region for each frame, which is a bitmap image, based on the detection flag acquired by the decoding unit 102 and added as metadata to the video (steps S206 and S207). The server 10 causes the mask processing unit 104 to perform mask processing on the detected specific object region (step S208).

[0097] The server 10 compresses and encodes the video, which has been subjected to the masking process in step S208 and includes each frame as a bitmap image, in a predetermined format such as H.264, for example, by the storage unit 105, stores the compressed and encoded video in, for example, the storage 1003, and registers the video in the SNS (step S209). Without being limited to this, the server 10 may store a video, which has not been compressed and includes each frame as a bitmap image, in the storage 1003. The storage unit 105 stores information indicating the position of each specific object region detected in each frame in the storage 1003, in association with each frame.

[0098] In step S203 described above, when it is determined that the access detected in step S200 is a request to acquire an image (step S203, "Acquire"), the process proceeds to step S210. In step S210, the storage unit 105 in the server 10 acquires a file in which the captured image requested to be acquired is stored, for example, from registered images stored in the storage 1003 and registered in the SNS.

[0099] In the next step S211, the server 10 determines whether or not a detection flag indicating that a specific object has been detected is included in the metadata stored in the file acquired in step S210 by the mask processing unit 104. If the mask processing unit 104 determines that the detection flag is not included (step S211, "No"), the process proceeds to step S213.

[0100] On the other hand, when the mask processing unit 104 determines that the detection flag is included (step S211, "Yes"), the server 10 shifts the process to step S212. In step S212, the mask processing unit 104 removes the mask applied to the captured image based on the information indicating the position of the specific object area associated with the captured image acquired in step S210. For example, if a mask is applied by encrypting the specific object area in the captured image, the mask processing unit 104 decrypts the encryption. This makes it possible to view the captured image in the state in which it is not masked and is received in step S204.

[0101] In the next step S213, the server 10 transmits the captured image acquired in step S210 (if the detection flag indicating that a specific object has been detected is not included) or the captured image unmasked in step S212 (if the detection flag is included) to the access source detected in step S200, via the communication unit 100. When the captured image is transmitted, the series of processes according to the flowchart in FIG. 13 is terminated.

[0102] If the access detected in step S200 is from an access source that has not been authenticated by the authentication process by the server 10 (for example, unauthorized access from the terminal device 41), the process of step S211 is not executed. Therefore, if a captured image registered in the SNS is acquired through unauthorized access, the acquired captured image remains masked, and the image included in the specific object area cannot be identified. Therefore, the spread of personal information included in the specific object area can be suppressed.

[0103] If the image requested to be acquired in step S203 is a video and the video is compressed and encoded, the server 10 causes the decoding unit 102 to decode the compressed code of the video acquired from the registered image in step S210 to generate a bitmap image for each frame, and acquires the detection flag added as metadata to the video. If the image requested to be acquired is a video stored in the storage 1003 without being compressed and encoded, the decoding process of the compressed code is omitted.

[0104] For a frame to which a detection flag set to a value of "1" has been added by the mask processing unit 104 (step S211), the server 10 removes the mask applied to the frame based on information indicating the position of a specific object area associated with the frame (step S212). The server 10 compresses and encodes each of the frames from which the mask has been removed by the mask processing unit 104 using a predetermined method such as the H.264 method, and transmits the compressed frame (step S213).

[0105] 17 is a sequence chart of an example showing the overall operation of the information processing system 1 according to the first embodiment. In the following, a description will be given on the assumption that the terminal device 30 transmits a still image captured by the imaging unit 3003 to the server 10.

[0106] When the installed SNS application is started in the terminal device 30 (step S300), the application unit 302 transmits account information to the server 10 via the authentication request unit 322, and requests authentication from the server 10 (step S301). If the account information transmitted from the terminal device 30 is correct, the server 10 notifies the terminal device 30 that authentication has been successful (step S302). This allows the terminal device 30 to use the SNS functions provided by the server 10.

[0107] After authentication of the terminal device 30, the imaging processing unit 309 of the imaging unit 3003 captures an image in response to a user operation in the terminal device 30, and a captured image is acquired (step S303). The terminal device 30 causes the imaging unit 3003 and the object detection unit 301 to detect a specific object in the captured image acquired by imaging (step S304). When a specific object is detected in the captured image, the object detection unit 301 adds a detection flag indicating that a specific object has been detected to the captured image (step S305), and passes the captured image with the detection flag added to the application unit 302. The captured image is passed to the encoding unit 320 in the application unit 302.

[0108] The terminal device 30 causes the encoding unit 320 to perform an encoding process on the captured image passed from the object detection unit 301 (step S306). This encoding process performs compression encoding, for example, by the JPEG format, and adds metadata including a detection flag to the captured image. The encoding unit 320 stores each piece of encoded data in a file in a predetermined format (for example, Exif format), and passes the file to the communication unit 303. The communication unit 303 transmits the file, in which each piece of data passed from the encoding unit 320 is stored, to the server 10 in response to, for example, a user operation, and uploads the captured image to the server 10 (step S307).

[0109] When the server 10 receives a file in which a captured image is stored and uploaded from the terminal device 30, the decoding unit 102 performs a decoding process on the received file (step S310). That is, the decoding unit 102 acquires metadata from the received file. At the same time, the decoding unit 102 decodes the compressed and encoded captured image stored in the file. The decoding unit 102 passes the decoded captured image to the analysis unit 103 and holds the captured image before decoding.

[0110] If the metadata acquired from the file includes a detection flag indicating that a specific object has been detected from the captured image stored in the file, the server 10 analyzes the decoded captured image passed from the decoding unit 102 by the analyzing unit 103 to detect all detectable specific objects included in the decoded captured image (step S311). The analyzing unit 103 passes position information indicating the positions of the specific objects detected from the decoded captured image to the mask processing unit 104. The mask processing unit 104 performs a mask process on the captured image before decoding (compressed and encoded captured image) held in the decoding unit 102 based on the position information passed from the analyzing unit 103 (step S312). The server 10 registers the masked captured image in the SNS.

[0111] 17, steps S320 to S324 show an example of processing in which the server 10 is accessed through legitimate access, the SNS is logged in, and a captured image to be registered in the SNS is obtained. For example, a terminal device 40 having an SNS app installed transmits account information to the server 10 from the SNS app, and an authentication is requested (step S320). If the account information transmitted from the terminal device 40 is correct, the server 10 notifies the terminal device 40 that authentication has been successful (step S321). This allows the terminal device 40 to use the SNS functions provided by the server 10.

[0112] For example, in response to a user operation, the terminal device 40 requests the server 10 to view the captured image registered in the SNS through an SNS application (step S322). In response to this request, the server 10 removes the mask applied to the captured image registered in the SNS (step S323). The server 10 transmits the unmasked captured image to the terminal device 40 (step S324). The terminal device 40 decodes the captured image, decodes the compressed code, and displays it on the display. Since the mask applied to the specific object in the captured image has been removed, the captured image can be viewed on the terminal device 40 so that the specific object can be identified.

[0113] 17, steps S330 to S332 show an example of processing in the case where the server 10 is accessed through unauthorized access, and a captured image to be registered in the SNS is acquired without logging in to the SNS. For example, it is assumed that the terminal device 41 accesses the server 10 without transmitting account information or making an authentication request to the server 10, and requests a captured image to be registered in the SNS without authentication by the server 10 (step S330).

[0114] The server 10 does not remove the mask applied to the captured image for a request from an unauthenticated access source (step S331). Therefore, the terminal device 41 acquires the masked captured image (step S332). Since the specific object remains masked in the captured image, the specific object cannot be identified in the captured image viewed on the terminal device 41. Therefore, even if the captured image is leaked from the terminal device 41 onto the Internet, for example, the personal information contained in the specific object is prevented from being spread.

[0115] In this way, in the first embodiment, depending on whether the access to the server 10 is legitimate or unauthorized, it is possible to control whether to remove the processing (masking) of the image to be registered or to leave it as it is, and it is possible to achieve both processed and unprocessed states for the image. In addition, in the case of unauthorized access, the image is left processed, so that it is possible to suppress the spread of, for example, personal information contained in the image before processing.

[0116] (Modification of the first embodiment) In the above description, the terminal device 30 detects whether or not at least one specific object is included in the captured image (FIG. 10, step S103), and when at least one specific object is included in the captured image, the server 10 analyzes the captured image to detect a specific object area (FIG. 13, step S207). This is not limited to this example, and the terminal device 30 may detect all specific objects that can be detected from the captured image.

[0117] That is, the terminal device 30 acquires specific object regions and their coordinates for all detected specific objects (for example, step S103 in FIG. 10). The terminal device 30 adds the coordinates of all acquired specific object regions to the captured image as metadata (for example, step S104 in FIG. 10).

[0118] In this case, since there is no need for server 10 to acquire the specific object region, for example, the process of step S207 in Fig. 13 (step S311 in Fig. 17) can be omitted. In server 10, mask processing unit 104 executes mask processing for each specific object region based on the coordinates of each specific object region included in the metadata added to the captured image (step S207 in Fig. 13, step S312 in Fig. 17).

[0119] According to the modified example of the first embodiment, each specific object area included in the captured image is acquired on the terminal device 30 side, so that it is possible to reduce the load on the server 10 compared to the processing of the first embodiment described above.

[0120] [Second embodiment] Next, a second embodiment will be described. In the above-mentioned first embodiment, the server 10 registers a captured image that has been subjected to a reversible masking process, and releases the masking process for the captured image in response to a legitimate access. In contrast, in the second embodiment, the server 10 registers a captured image without performing a masking process, and performs a masking process on the captured image in response to a request for acquiring the captured image through an unauthorized access. In the second embodiment, a non-reversible method can be used for the masking process.

[0121] Fig. 18 is a flowchart showing an example of processing in the server 10 according to the second embodiment. In Fig. 18, processing common to the flowchart in Fig. 13 described above is indicated by the same reference numerals. In addition, the processing of the terminal device 30 when registering a captured image in an SNS provided by the server 10 can be directly applied to the processing described in Fig. 10.

[0122] In FIG. 18, in step S200, the server 10 detects access via the network 2 using the communication unit 100, and in the next step S201, the authentication unit 101 determines whether the access detected in step S200 is access from an access source that has already received an authentication request.

[0123] When the authentication unit 101 determines that the access detected in step S200 is an access for which an authentication request has already been received (step S201, "Yes"), the server 10 shifts the process to step S202. In step S202, the authentication unit 101 determines whether or not authentication has been successful for the received authentication request. When the authentication unit 101 determines that authentication has not been successful for the received authentication request (step S202, "NG"), the authentication unit 101 ends the series of processes according to the flowchart of FIG. 18.

[0124] On the other hand, if the authentication unit 101 determines that the authentication for the received authentication request has been successful (step S202, "OK"), the server 10 shifts the process to step S203. In step S203, the server 10 determines, for example, by the communication unit 100, whether the access detected in step S200 is a request for image registration or acquisition. If the communication unit 100 determines that the access detected in step S200 is a request for image registration (step S203, "Register"), the server 10 shifts the process to step S204. On the other hand, if the communication unit 100 determines that the access detected in step S200 is a request for image acquisition (step S203, "Acquire"), the server 10 shifts the process to step S210.

[0125] In step S204, the server 10 receives the file storing the image transmitted from the access source detected in step S200, via the communication unit 100. This access source is, for example, the terminal device 30, and the communication unit 100 receives the file storing the captured image transmitted in step S106 in the flowchart of FIG.

[0126] In the next step S205, the server 10 extracts the compressed and encoded captured image from the file received in step S204 and decodes the captured image by using the decoding unit 102. The decoding unit 102 also holds the compressed and encoded captured image stored in the received file. The decoding unit 102 also obtains metadata from the file.

[0127] In the next step S206, the server 10 determines, via the decoding unit 102, whether or not the metadata acquired in step S205 includes a detection flag indicating that a specific object has been detected. If the decoding unit 102 determines that the detection flag is not included (step S206, "No"), the process proceeds to step S209'. On the other hand, if the decoding unit 102 determines that the detection flag is included (step S206, "Yes"), the process proceeds to step S207.

[0128] In step S207, the server 10 analyzes the captured image decoded in step S205 by the analysis unit 103, and detects a specific object region from the captured image. The analysis unit 103 stores the coordinates of the detected specific object region in association with the captured image. When the specific object region is detected in step S207, the process proceeds to step S209'. In the second embodiment, the mask process for the detected specific object region in step S208 in Fig. 13 is not executed when the captured image is registered.

[0129] In the next step S209', the server 10 causes the memory unit 105 to store the captured image (image file) received in step S204 and the coordinates of the specific object area associated with the captured image in step S207 in the storage 1003. When the captured image is registered in the SNS, the series of processes according to the flowchart in Fig. 18 is terminated.

[0130] In the above-mentioned step S203, when it is determined that the access detected in step S200 is a request for acquiring an image (step S203, "Acquire"), the process proceeds to step S210. In step S210, the server 10 acquires, by the memory unit 105, a file in which the captured image requested to be acquired is stored, for example, from registered images stored in the storage 1003 and registered in the SNS.

[0131] When the registered image is acquired in step S210, the process proceeds to step S213. That is, in the second embodiment, since the registered image is not masked, the determination process based on the detection flag in step S211 in Fig. 13 and the mask removal process in step S212 are omitted. In step S213, the server 10 transmits the captured image acquired in step S210 to the access source whose access was detected in step S200, via the communication unit 100. When the captured image is transmitted, the series of processes according to the flowchart in Fig. 18 is terminated.

[0132] In the above-mentioned step S201, if the authentication unit 101 determines that the access detected in step S200 is not an access for which an authentication request has already been received (step S201, "No"), the server 10 shifts the process to step S220. In step S220, the server 10 determines, for example, by the communication unit 100, whether or not the access detected in step S200 is a request for obtaining an image. If the communication unit 100 determines that the access is not a request for obtaining an image (step S220, "No"), the series of processes according to the flowchart in FIG. 18 is terminated.

[0133] On the other hand, when the server 10 determines through the communication unit 100 that the access is a request to obtain an image (step S220, "Yes"), the server 10 shifts the process to step S221. In step S221, the server 10 obtains, through the mask processing unit 104, the captured image requested by the access source of the access detected in step S200, from among the captured images registered in step S209'. The server 10 performs mask processing on an area of ​​the obtained captured image corresponding to a specific object of the captured image, based on the coordinates of the specific object associated with the captured image. In the next step S222, the server 10 transmits the captured image subjected to the mask processing to the access source of the access detected in step S200, through the communication unit 100.

[0134] When the captured image on which the masking process has been performed is transmitted in step S222, the series of processes according to the flowchart of FIG. 18 ends.

[0135] In the second embodiment, it is preferable to perform a lossy mask process on the captured image in step S221. An example of the lossy mask process is a process of replacing an area of ​​the captured image corresponding to a specific object area with another image such as a solid image with a uniform color or a predetermined icon image. Not limited to this, the entire captured image may be replaced with a solid image or the like. The same applies when the captured image is a moving image.

[0136] In this way, by irreversibly masking the area corresponding to the specific object area of ​​the captured image, the captured image acquired by unauthorized access to the SNS is masked, and the image included in the specific object area cannot be identified. Therefore, the spread of personal information included in the specific object area can be suppressed. Furthermore, by irreversibly masking the area of ​​the captured image corresponding to the specific object area, it becomes impossible for the receiver of the captured image to identify the image included in the specific object area. Therefore, the spread of personal information included in the specific object area can be more effectively suppressed.

[0137] In the second embodiment, since the registered captured image itself is not masked, if the registered captured image is accessed by some method without logging in to the SNS, the original captured image may be obtained. Therefore, it is preferable to take measures against unauthorized access.

[0138] Fig. 19 is a sequence chart of an example showing the overall operation of the information processing system 1 according to the second embodiment. In the following description, it is assumed that the terminal device 30 transmits a captured image, which is a still image captured by the imaging unit 3003, to the server 10. In Fig. 19, among the processes from acquiring a captured image in the terminal device 30 to registering the captured image in the server 10, the processes of steps S300 to S310 are the same as those described using Fig. 17, and therefore description thereof will be omitted here. In the process of Fig. 19, the mask process of step S312 in Fig. 17 is omitted.

[0139] In Fig. 19, steps S320 to S324 show an example of processing in the case where, for example, the terminal device 40 accesses the server 10 by regular access, logs into the SNS, and acquires a captured image to be registered in the SNS. In the case of the second embodiment, the registered captured image is not subjected to mask processing, so the processing in Fig. 19 omits the mask removal processing in step S323 in Fig. 17. In Fig. 19, steps S320 to S322 are the same as the processing in steps S320 to S322 in Fig. 17, so description thereof will be omitted here.

[0140] In FIG. 19, in step S324', the server 10 transmits the captured image requested by the terminal device 40 through the communication unit 100 (step S324). The terminal device 40 decodes the captured image, decodes the compressed code, and displays the image on the display. Since the captured image has not been subjected to mask processing, the captured image can be viewed on the terminal device 40 so that specific objects can be identified.

[0141] 19, steps S340 to S342 show an example of processing in the case where the server 10 is accessed through unauthorized access, and a captured image to be registered in the SNS is acquired without logging in to the SNS. For example, it is assumed that the terminal device 41 accesses the server 10 without transmitting account information or making an authentication request to the server 10, and requests a captured image to be registered in the SNS without authentication by the server 10 (step S340).

[0142] In response to the request from the terminal device 41 in step S340, the server 10 performs irreversible mask processing on the requested captured image by the mask processing unit 104 (step S341). Therefore, the terminal device 41 acquires the captured image that has been irreversibly masked (step S342). Since the specific object in the captured image is irreversibly masked, the specific object cannot be identified in the captured image viewed on the terminal device 41. Therefore, even if the captured image is leaked from the terminal device 41 to the Internet, for example, personal information contained in the specific object is prevented from being spread.

[0143] In this way, in the second embodiment, whether to process (mask) the requested image or not can be controlled depending on whether the access to the server 10 is legitimate or unauthorized, and both processed and unprocessed states of the image can be achieved. In addition, in response to unauthorized access, the image is irreversibly processed, so that the spread of, for example, personal information contained in the image before processing can be suppressed.

[0144] In addition, in the second embodiment as well, the process of detecting all specific objects that can be detected from a captured image in the terminal device 30, which was described in the modified example of the first embodiment described above, can be applied as a modified example.

[0145] [Third embodiment] Next, a third embodiment will be described. In the above, the first and second embodiments are applied to a single service by the server 10, but the present disclosure is not limited to this example. That is, the present disclosure is also applicable to a plurality of services that are linked together.

[0146] FIG. 20 is a diagram showing an example of the configuration of an information processing system according to the third embodiment. In FIG. 20, an information processing system 1′ according to the third embodiment includes a server 10 and a server 11. The server 10 provides the SNS (referred to as service #A) described in the first or second embodiment. In the following, the server 10 provides the SNS described in the first embodiment as service #A. The server 11 provides service #B. Service #B is available upon performing authentication processing, and may be an SNS or another service. Here, service #B is assumed to be an SNS different from service #A.

[0147] Service #A and service #B are linked, so that, for example, a user who has logged in to server 11 and is able to use service #B can use service #A provided by server 10 without explicitly logging in to server 10.

[0148] As an example, it is assumed that a user 31 accesses the server 10 from a terminal device 30, logs in to service #A, and registers a captured image captured by the terminal device 30 in service #A. The terminal device 30 registers the captured image in the storage 1003 of the server 10, for example, by the processes of steps S300 to S312 in Fig. 17. At this time, the server 10 performs a mask process on a specific object region included in the captured image, stores the captured image in the storage 1003, and registers the captured image in service #A.

[0149] Here, it is assumed that the terminal device 40 is equipped with an SNS application (B) for using service #B. The functional configuration of the terminal device 40 is equivalent to the configuration described using FIG. 8, and the function of this SNS application (B) is also equivalent to the function of the application unit 302 in FIG. 8. The user 42 starts up the SNS application (B) on the terminal device 40, transmits account information for service #B to the server 11, and requests authentication. If the authentication by the server 11 is successful, the terminal device 40 logs in to service #B. This makes it possible to use the functions of service #B from the terminal device 40.

[0150] Here, service #B cooperates with service #A, and due to the cooperation between these services, terminal device 40 can use the functions of service #A by logging in to service #B. In this case, terminal device 40 can obtain captured images stored in storage 1003 and registered in service #A via service #B.

[0151] When the server 10 receives an acquisition request for a captured image registered in service #A via service #B (server 11), the server 10 considers that the terminal device 40 has transmitted the acquisition request for the captured image after passing through authentication processing, for example, in steps S320 and S321 in Fig. 17. Therefore, the server 10 removes the mask covering the specific object area of ​​the captured image according to the processing in step S322 in Fig. 17, and transmits the unmasked captured image to the terminal device 40 via service #B (server 11). This allows the terminal device 40 to view the captured image in a manner that allows the specific object to be identified.

[0152] Next, a case will be described where, for example, a captured image registered in service #A is acquired from the terminal device 41 without logging in to either service #A or service #B. In this case, as described in steps S330 to S332 of FIG. 17, the server 10 does not remove the mask applied to the captured image for a request from an unauthenticated access source. Therefore, the terminal device 41 acquires a masked captured image, and a specific object cannot be identified in the captured image viewed on the terminal device 41. Therefore, even if the captured image is leaked from the terminal device 41 onto the Internet, for example, personal information included in the specific object is prevented from being spread.

[0153] In this manner, the present disclosure is also applicable to cases where multiple services work together.

[0154] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0155] The present technology can also be configured as follows. (1) An image sensor that converts the received light into an image; an image processing unit that detects a specific object in the image; an output interface for outputting the image; A solid-state imaging element including: a first communication unit that transmits an image output from the output interface; A terminal device comprising: A second communication unit; A processor; An information processing device comprising: having The processor, receiving an image transmitted from the terminal device by the second communication unit; When the image received by the second communication unit is an image in which the specific object is detected by the image processing unit, a masking process is performed to mask the specific object. Information processing system. (2) The processor, storing the image in which the specific object is masked in a storage unit; When an access request for the image stored in the storage unit is received, it is determined whether or not the sender of the access request is permitted to access the image; If it is determined that access to the image is permitted, the mask is removed. The information processing system according to (1) above. (3) The processor, The specific object is reversibly masked by the mask processing. The information processing system according to (1) or (2). (4) The processor, The specific object is masked by encrypting a predetermined area including the specific object in the image in which the specific object is detected. An information processing system according to any one of (1) to (3). (5) The processor, storing the received image in a storage unit; When an access request for the image stored in the storage unit is received, it is determined whether or not the sender of the access request is permitted to access the image; When it is determined that access to the image is not permitted, a masking process is performed to mask the specific object of the image. The information processing system according to (1) above. (6) The processor, The specific object is irreversibly masked by the mask processing. The information processing system according to (5) above. (7) The image processing unit includes: adding detection information indicating that the specific object has been detected to the image output from the image sensor when the specific object is detected from the image output from the image sensor; The first communication unit is Transmitting the image to which the detection information has been added; The processor, When the detection information is added to the image received by the second communication unit, the specific object included in the image is masked. An information processing system according to any one of (1) to (6). (8) The processor, When the detection information is added to the image received by the second communication unit, position information indicating a position of the specific object in the image is obtained, and the specific object is masked based on the position information. The information processing system according to (7) above. (9) The image processing unit includes: Detecting the specific object from the image output from the image sensor, and acquiring position information indicating a position of the detected specific object on the image; The first communication unit is adding the position information as the detection information to the image output from the output interface and transmitting the image; The processor, The specific object included in the image is masked based on the position information added to the image communicated by the second communication unit. The information processing system according to (7) above. (10) The information processing device includes: Includes multiple information processing devices that operate in cooperation An information processing system according to any one of (1) to (9) above. (11) A conversion step in which an image sensor included in a fixed imaging element of the terminal device converts received light into an image; a detection step in which an image processing unit included in the fixed imaging element detects a specific object in the image; an output step in which an output interface included in the fixed imaging device outputs the image; a transmitting step in which a first communication unit included in the terminal device transmits the image outputted in the output step; A receiving step in which a processor included in the information processing device receives the image transmitted from the terminal device by a second communication unit; a processing step of the processor masking the specific object when the image received by the receiving step is an image in which the specific object is detected by the detecting step; An information processing method comprising the steps of: (12) An image sensor that converts the received light into an image; an image processing unit for detecting a specific object from the image; an output interface for outputting the image; A solid-state imaging element including: a communication unit for transmitting an image output from the output interface; Equipped with The image processing unit includes: adding detection information indicating that the specific object has been detected in the image when the specific object has been detected from the image; The communication unit is Transmitting the image to which the detection information has been added Terminal device. (13) The image processing unit includes: Detecting the specific object from the image output from the image sensor, and acquiring position information indicating a position of the detected specific object on the image; The communication unit is The position information is added as the detection information to the image output from the output interface and transmitted. The terminal device according to (12) above. (14) A processor; a communication unit that detects a specific object in an image obtained by converting the light received by the image sensor and communicates with a terminal device that transmits the image; Equipped with The processor, receiving the image transmitted from the terminal device by the communication unit; When the image received by the communication unit is an image in which the specific object is detected by the terminal device, a masking process is performed to mask the specific object. Information processing device. (15) The processor, storing the masked image of the specific object in a storage unit; When an access request for the image stored in the storage unit is received, it is determined whether or not the sender of the access request is permitted to access the image; If it is determined that access to the image is permitted, the mask is removed. The information processing device according to (14). (16) The processor, The specific object is reversibly masked by the mask processing. The information processing device according to (14) or (15). (17) The processor, The specific object is masked by encrypting a predetermined area including the specific object in the image in which the specific object is detected. 17. The information processing device according to any one of (14) to (16). (18) The processor, storing the received image in a storage unit; When an access request for the image stored in the storage unit is received, it is determined whether or not the sender of the access request is permitted to access the image; When it is determined that access to the image is not permitted, the specific object of the image is masked. The information processing device according to (14). (19) The processor, The specific object is irreversibly masked by the mask processing. The information processing device according to (18). (20) The processor, When detection information indicating that the specific object has been detected is added to the image received by the communication unit, the specific object included in the image is masked. 20. The information processing device according to any one of (14) to (19). (twenty one) The processor, When the detection information is added to the image received by the communication unit, position information indicating a position of the specific object in the image is obtained, and the specific object is masked based on the position information. The information processing device according to (20). (twenty two) The processor, The specific object included in the image is masked based on position information indicating a position of the specific object on the image, which is added as the detection information to the image communicated by the communication unit. The information processing device according to (20). [Explanation of symbols]

[0156] 1. Information Processing Systems 2 Network 10,11 Server 30, 40, 41 Terminal equipment 50a, 50a', 50b, 50b' Images 51a,51b,52a,52b rectangular area 100 Communications Department 101 Authentication Department 102 Decoding section 103 Analysis Department 104 Mask processing section 105 Storage section 300 Imaging processing section 301 Object Detection Unit 302 Application Department 303 Communications Department 320 Encoding section 322 Authentication Request Section 3003 Imaging unit 3100 Pixel unit 3101 Signal processing section 3102 Memory section 3103 I / F section 3104 Image processing unit 3105 Element control unit

Claims

1. an image sensor that converts the received light into an image and outputs the image; an image processing unit that detects a specific object including an open window in the captured image; an output interface for outputting the captured image; A solid-state imaging element including: a first communication unit that transmits the captured image output from the output interface; A terminal device comprising: A second communication unit; A processor; An information processing device comprising: having The information processing device includes: receiving the captured image transmitted from the terminal device as a received image by the second communication unit in accordance with an instruction from the processor; when the received image received by the second communication unit is an image in which the specific object has been detected by the image processing unit, and an access request for the received image received by the second communication unit is received from a source other than the terminal device, determining by the processor whether or not the source is permitted to access the received image; when the processor determines that the source is not permitted to access the received image, a masking process is performed on the received image to mask the specific object by replacing it with another image, and the image obtained by performing the masking process on the received image is transmitted to the source; transmitting the received image to the source without performing the masking process when the processor determines that the source is permitted to access the received image; Information processing system.

2. The processor, storing the received image in a storage unit; when receiving an access request for the received image stored in the storage unit, determining whether or not a sender of the access request is permitted to access the received image; When it is determined that the sender is not permitted to access the received image, the masking process is performed. The information processing system according to claim 1 .

3. The image processing unit includes: when the specific object is detected from the captured image output from the image sensor, adding detection information indicating that the specific object has been detected to the captured image; The first communication unit is Transmitting the captured image to which the detection information has been added; The processor, When the detection information is added to the received image received by the second communication unit, the masking process is performed on the specific object included in the received image. The information processing system according to claim 1 .

4. The processor, when the detection information is added to the received image received by the second communication unit, acquiring position information indicating a position of the specific object in the received image, and performing the masking process of masking the specific object in the received image based on the position information; The information processing system according to claim 3 .

5. The image processing unit includes: Detecting the specific object from the captured image output from the image sensor, and acquiring position information indicating a position of the detected specific object on the captured image; The first communication unit is adding the position information as the detection information to the captured image output from the output interface and transmitting the image; The processor, performing the masking process of masking the specific object included in the received image based on the position information added to the received image received by the second communication unit; The information processing system according to claim 3 .

6. The information processing device includes: Includes multiple information processing devices that operate in cooperation The information processing system according to claim 1 .

7. A conversion step in which an image sensor included in a fixed imaging element of the terminal device converts received light into an image to obtain a captured image; a detection step in which an image processing unit included in the fixed imaging element detects a specific object including an open window from the captured image; an output step in which an output interface included in the fixed imaging element outputs the captured image; a transmitting step in which a first communication unit included in the terminal device transmits the captured image outputted in the output step; a receiving step in which a processor included in the information processing device receives the captured image transmitted from the terminal device as a received image via a second communication unit; a processing step of the processor performing a masking process to mask the specific object when the image received in the receiving step is an image in which the specific object is detected in the detecting step; a determination step in which, when the received image received in the receiving step is an image in which the specific object has been detected in the detecting step and an access request for the received image received by the second communication unit has been received from a source other than the terminal device, the processor determines whether or not the source is permitted to access the received image; a transmitting step of transmitting to the source an image subjected to the masking process, which replaces the specific object of the received image with another image, when the processor determines that the source is not permitted to access the received image, and transmitting to the source the received image without performing the masking process, when the processor determines that the source is permitted to access the received image; An information processing method comprising the steps of:

8. A processor; a communication unit that detects a specific object including an open window from a captured image obtained by converting light received by the image sensor, and communicates with a terminal device that transmits the captured image; Equipped with The processor, receiving the captured image transmitted from the terminal device as a received image by the communication unit; when the received image received by the communication unit is an image in which the specific object has been detected by the terminal device and an access request for the received image received by the communication unit has been received from a source other than the terminal device, determining whether or not the source is permitted to access the received image; When the processor determines that the transmission source is not permitted to access the received image, the processor transmits to the transmission source an image obtained by performing a masking process on the received image, which masks the specific object of the received image by replacing it with another image; When the processor determines that the source is permitted to access the received image, the processor transmits the received image to the source without performing the masking process.

9. The processor, The received image is stored in a storage unit. when receiving an access request for the received image stored in the storage unit, determining whether or not a sender of the access request is permitted to access the received image; performing the masking process when it is determined that the transmission source is not permitted to access the received image; The information processing device according to claim 8.

10. The processor, when detection information indicating that the specific object has been detected is added to the received image received by the communication unit, the masking process is performed on the specific object included in the received image; The information processing device according to claim 8.

11. The processor, when the detection information is added to the received image received by the communication unit, acquiring position information indicating a position of the specific object in the received image, and performing the masking process on the specific object based on the position information; The information processing device according to claim 10.

12. The processor, performing the masking process on the specific object included in the received image based on position information indicating a position of the specific object on the received image, the position information being added as the detection information to the received image communicated by the communication unit; The information processing device according to claim 10.

Citation Information

Patent Citations

  • Face extracting device, face extracting method and recording medium for face extraction program

    JP2001222719A

  • Video distribution system

    JP2006319775A

  • Imaging apparatus, data structure of image file

    JP2009033738A

  • Image processing apparatus, camera device, communication system, image processing method, and program

    JP2009194687A

  • Image processing system, data management method and program

    JP2015184909A