Information processing system, information processing method, program, and user interface

Abstract

The present invention guarantees that captured image data is not altered while not increasing cost. To this end, this information processing system is provided with: an array sensor including a pixel array in which a plurality of pixels having light reception elements of visible light or non-visible light are arranged one or two-dimensionally, a hash value generation unit which generates a hash value from captured image data obtained on the basis of optical-to-electrical conversion in the pixel array, and an encryption processing unit which executes an encryption process for the hash value; an acquisition unit which acquires the captured image data and the encrypted hash value; a decryption processing unit which decrypts the acquired encrypted hash value; a hash value calculation unit which calculates a hash value from the acquired captured image data; a hash value comparison unit which compares the decrypted hash value with the calculated hash value; and an alteration determination unit which determines, on the basis of the comparison result of the hash values, whether the acquired captured image data has been altered.

Description

[Technical field]

[0001] The present technology relates to an information processing system, an information processing method, a program, and a user interface, and in particular to a technology for detecting whether data has been tampered with. [Background technology]

[0002] 2. Description of the Related Art Image processing technology has been developing, and as a result, it has become possible to masquerade as another person by tampering with a captured image of the face or body of a person. In addition, by creating artificial human images and voices and having a machine communicate interactively with them, there is a possibility that a problem may arise in which the person communicating with them will mistake them for a real person and communicate with them without realizing that they are a machine. To address such problems, for example, Patent Document 1 listed below describes detecting tampering with captured image data based on electronic signature information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-184198 A Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, in order to prevent tampering with captured image data, it is conceivable to make the transmission path through which the captured image data is transmitted highly secure; however, it is disadvantageous in terms of cost to increase the security of all transmission paths. Specifically, a hash value is generated based on the output from an image sensor, but if the output from the image sensor is leaked at a stage before the hash value is generated, it cannot be guaranteed that the captured image data has not been tampered with.

[0005] Therefore, an object of the present technology is to ensure that captured image data has not been tampered with without increasing costs. [Means for solving the problem]

[0006] The information processing system according to the present technology includes an array sensor having a pixel array in which a plurality of pixels, each having a light receiving element for visible light or invisible light, are arranged one-dimensionally or two-dimensionally, a hash value generation unit that generates a hash value from captured image data obtained based on photoelectric conversion in the pixel array, and an encryption processing unit that performs encryption processing on the hash value, an acquisition unit that acquires the captured image data and the encrypted hash value, a decryption processing unit that decrypts the acquired encrypted hash value, a hash value calculation unit that calculates a hash value from the acquired captured image data, a hash value comparison unit that compares the decrypted hash value with the calculated hash value, and a tampering determination unit that determines whether the acquired captured image data has been tampered with based on a comparison result of the hash values. That is, the generation of the hash value and the encryption process are performed within the array sensor.

[0007] The above-mentioned information processing system may include a biometric analysis processing unit that performs analysis to extract biometric information of the subject based on the acquired captured image data, and a biometric determination unit that determines whether the subject is a living organism based on the analysis results of the biometric analysis processing unit. This makes it possible to determine whether the subject is alive or not.

[0008] The above-mentioned information processing system may include an individual analysis processing unit for analyzing individual features of the subject contained in the acquired captured image data, and an individual determination unit for determining whether or not the subject can be recognized as an individual based on the analysis results of the individual analysis processing unit. This makes it possible to obtain information for identifying the individual subject.

[0009] The above-mentioned information processing system may further include an output control unit that performs output control for notifying the determination result of the tamper determination unit. This allows the user to recognize the tampering determination result. In the above-described information processing system, the subject in the captured image data may be a subject from which biological information cannot be extracted.

[0010] The information processing system may further include an output control unit that performs output control for notifying the user of the determination result of the liveness determination unit. This allows the user to recognize the liveness determination result.

[0011] The information processing system may further include an output control unit that performs output control for notifying the user of the determination result of the individual determination unit. This allows the user to recognize the result of the individual determination.

[0012] The above-mentioned information processing system may be configured to include an imaging device and an information processing device, wherein the imaging device has the array sensor, and the information processing device may have the acquisition unit, the decoding processing unit, the hash value calculation unit, the hash value comparison unit, the tampering determination unit, and the output control unit. For example, the present invention can be applied to a surveillance camera system that is made up of an imaging device and an information processing device.

[0013] In the above-mentioned information processing system, the output control unit may execute control to display and output the acquired captured image data, and control to display and output the judgment result of the tamper judgment unit on the acquired captured image data. One example of such an information processing system is one that performs user interface control to display a received captured image while indicating whether or not the image has been tampered with.

[0014] The output control unit in the above-described information processing system may perform control to display and output a determination result made by the tamper determination unit without displaying the acquired captured image data. An example of such an information processing system is a text chat system in which captured images are not displayed.

[0015] The output control unit in the above-mentioned information processing system may perform control to present a chat screen and control to display and output on the chat screen for each chat participant the judgment result of the tampering judgment unit regarding the captured image data of the chat participants received from the imaging device used by the chat participants. In a text chat system where captured images are not displayed, an indicator of whether each participant is trustworthy is displayed.

[0016] The above-mentioned information processing system may be configured to include an imaging device and an information processing device, wherein the imaging device has the array sensor and the output control unit, and the information processing device may have an authentication processing unit that performs authentication processing, the acquisition unit, the decryption processing unit, the hash value calculation unit, the hash value comparison unit, and the tampering determination unit. As an example of such an information processing system, a system in which an authentication result or the like is notified in an information processing device (imaging device) that has transmitted captured image data is considered.

[0017] The output control unit in the above-mentioned information processing system may control the display and output of the judgment result of the tampering judgment unit obtained from the information processing device, and the authentication processing unit may perform the authentication processing based on the judgment result of the tampering judgment unit. That is, in the device (imaging device) that transmitted the captured image data, the user can recognize whether the transmitted captured image data has reached the server device (information processing device) that is the receiving device without being tampered with.

[0018] The information processing device in the above-mentioned information processing system may include a biometric analysis processing unit that performs analysis to extract biometric information of the subject based on the acquired captured image data, and a biometric determination unit that determines whether the subject is a living body or not based on the analysis result of the biometric analysis processing unit, and the output control unit may control the display and output of the determination result of the biometric determination unit, and the authentication processing unit may perform the authentication process based on the determination result of the biometric determination unit. That is, in addition to the tampering determination process, the image processing device extracts biometric information from the received captured image data and performs a process of determining whether or not the subject is a living body based on the extraction result.

[0019] The information processing system described above may include an individual analysis processing unit for analyzing individual features of the subject included in the acquired captured image data; an individual determination unit for determining whether or not the subject can be recognized as an individual based on an analysis result of the individual analysis processing unit; and a state determination unit for determining, based on the determination result on the presence or absence of tampering, the determination result on whether or not the subject is a living body, and the determination result on whether or not the individual can be recognized, a non-secure state in which it is not guaranteed that the acquired captured image data has not been tampered with, a first secured state in which it is guaranteed that the acquired captured image data has not been tampered with, a second secured state in which it is guaranteed that the acquired captured image data has not been tampered with and the subject is determined to be a living body, or a third secured state in which it is guaranteed that the acquired captured image data has not been tampered with, the subject is determined to be a living body, and the individual recognition is possible. This makes it possible to carry out processing according to the degree of security assurance.

[0020] The above information processing system may further include an output control unit that performs display output according to the non-secured state, the first secured state, the second secured state, and the third secured state. This allows the user to check the degree of guarantee.

[0021] The living body determination section in the information processing system may determine whether or not the subject is a living body by determining the presence or absence of a change in a biological signal of the subject obtained as a result of the analysis. For example, a change in a biological signal is extracted by analyzing a plurality of captured image data.

[0022] The information processing method of the present technology is an information processing device that executes the following processes from an array sensor having a pixel array in which multiple pixels having visible light or invisible light receiving elements are arranged in one or two dimensions, a hash value generation unit that generates a hash value from captured image data obtained by photoelectric conversion, and an encryption processing unit that performs encryption processing on the hash value: a process of acquiring the captured image data and the encrypted hash value, a process of decrypting the acquired encrypted hash value, a process of calculating a hash value from the acquired captured image data, a process of comparing the decrypted hash value with the calculated hash value, and a process of determining whether the acquired captured image data has been tampered with based on the comparison result of the hash values.

[0023] The program of the present technology causes an information processing device to execute the following processes from an array sensor having a pixel array in which multiple pixels having visible light or invisible light receiving elements are arranged in one or two dimensions, a hash value generation unit that generates a hash value from captured image data obtained by photoelectric conversion, and an encryption processing unit that performs encryption processing on the hash value: a process of acquiring the captured image data and the encrypted hash value, a process of decrypting the acquired encrypted hash value, a process of calculating a hash value from the acquired captured image data, a process of comparing the decrypted hash value with the calculated hash value, and a process of determining whether the acquired captured image data has been tampered with based on the comparison result of the hash values. This allows generation of a hash value and encryption processing to be performed within the array sensor.

[0024] The user interface of the present technology notifies the result of a determination process as to whether or not the captured image data has been tampered with, which is performed based on a comparison result between a hash value obtained by decrypting an encrypted hash value obtained from an array sensor having a pixel array in which multiple pixels having visible light or invisible light receiving elements are arranged in one or two dimensions, a hash value generation unit that generates a hash value from captured image data obtained by photoelectric conversion, and an encryption processing unit that performs encryption processing on the hash value, and a hash value calculated from the captured image data obtained from the array sensor. [Brief description of the drawings]

[0025] [Figure 1] FIG. 1 is a schematic explanatory diagram illustrating an example of the configuration of an information processing system according to a first embodiment. [Diagram 2] 2 is a block diagram illustrating an example of the internal configuration of a user terminal according to the first embodiment; FIG. [Diagram 3] FIG. 1 is a schematic diagram for explaining a flow when using a video chat system according to a first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of an incoming call screen in the video chat system according to the first embodiment; [Diagram 5] FIG. 2 is a diagram showing an example of a call screen in the video chat system according to the first embodiment; [Figure 6] FIG. 13 is a diagram illustrating an example of a second guarantee icon according to the embodiment. [Figure 7] FIG. 13 is a diagram illustrating an example of a third guarantee icon according to the embodiment. [Figure 8] FIG. 13 is a diagram showing an example of a non-guaranteed icon according to an embodiment. [Figure 9] 5 is a flowchart showing a process executed by a sender's user terminal in the video chat system according to the first embodiment; [Figure 10] 5 is a flowchart showing a process executed by a receiver's user terminal in the video chat system according to the first embodiment; [Figure 11]FIG. 11 is a block diagram illustrating an example of the internal configuration of a user terminal according to a second embodiment. [Figure 12] FIG. 13 is a diagram showing an example of a method for generating an encryption filter (encryption key) based on a photoelectric random number. [Figure 13] 13 is a conceptual diagram for explaining encryption of a read signal by amplitude control in the second embodiment. FIG. [Figure 14] 13 is a flowchart showing a process executed by a sender's user terminal in a video chat system according to a second embodiment. [Figure 15] 13 is a flowchart showing encryption processing of captured image data in the second embodiment. [Figure 16] 13 is a flowchart showing a process executed by a receiver's user terminal in a video chat system according to a second embodiment. [Figure 17] 13 is a flowchart showing a process executed by a receiving user terminal in a video chat system according to a third embodiment. [Figure 18] 13 is a flowchart showing a process executed by a receiving user terminal in a video chat system according to a third embodiment. [Figure 19] FIG. 13 is a diagram showing an application example of the third embodiment, showing an example of a text chat screen. [Figure 20] FIG. 13 is a schematic explanatory diagram showing a configuration example of an information processing system according to a fourth embodiment. [Figure 21] FIG. 13 is a block diagram showing an example of the internal configuration of a user terminal according to the fourth embodiment. [Figure 22] FIG. 13 is a block diagram showing an example of the internal configuration of a server device according to a fourth embodiment. [Figure 23] FIG. 13 is a diagram illustrating an example of a login screen in an information processing system according to a fourth embodiment. [Figure 24] 13 is a flowchart showing a process executed by a user terminal in an information processing system according to a fourth embodiment. [Diagram 25]13 is a flowchart showing a process executed by a server device in an information processing system according to a fourth embodiment. [Figure 26] 13 is a diagram illustrating an example of a sign-up screen in the information processing system according to the fourth embodiment. [Figure 27] 13 is a flowchart showing a process executed by a user terminal in an information processing system as a modified example. [Figure 28] FIG. 1 illustrates an unaltered image displayed on an auction site. [Figure 29] FIG. 13 shows an unaltered image displayed on an accommodation search site. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] The embodiments will be described below in the following order. <1. First embodiment> <1-1. System configuration> <1-2. User terminal configuration> <1-3. Overall flow> <1-4. Processing on the sending side> <1-5. Processing on the receiving side> <2. Second embodiment> <2-1. System configuration> <2-2. Signals to be encrypted> <2-3. Processing on the sending side> <2-4. Processing on the receiving side> <3. Third embodiment> <3-1. Processing on the sending side> <3-2. Processing on the receiving side> <3-3. Application Examples> <4. Fourth embodiment> <4-1. System configuration> <4-2.Login screen> <4-3. User terminal processing> <4-4. Processing of server device> <5. Modifications> <6. Summary> <7. This Technology>

[0027] <1. First embodiment> <1-1. System configuration> FIG. 1 is a block diagram showing a schematic configuration example of an information processing system 1 according to the first embodiment. In the first embodiment, the information processing system 1 is exemplified as a video chat system.

[0028] The information processing system 1 includes a user terminal 2 that is used by a user to utilize the video chat system, a network 3, and a server device 4. The user terminal 2 is an imaging device and an information processing device according to the present technology. Specifically, the user terminal 2 is a form of an imaging device that captures an image using an image sensor to obtain image data (captured image data) as digital data. The user terminal 2 is also a form of an information processing device that executes a process for verifying that captured image data acquired from another user terminal 2 via the network 3 has not been tampered with.

[0029] In the following description, the user terminal 2 that captures images using an image sensor and transmits captured image data will be referred to as user terminal 2A. Also, the user terminal 2 that receives captured image data from user terminal 2A will be referred to as user terminal 2B. In the information processing system 1 as a video chat system, captured image data is sent and received bidirectionally. Therefore, one user terminal 2 can be regarded as both a user terminal 2A and a user terminal 2B.

[0030] In this embodiment, the target data for the electronic signature is image data captured by this user terminal 2A.

[0031] When transmitting captured image data to the user terminal 2B, the user terminal 2A generates a private key for an electronic signature and a public key based on the private key, calculates a hash value for the captured image data to be transmitted, and encrypts the hash value with the private key. The user terminal 2A then transmits the encrypted hash value, the captured image data to be transmitted, and the public key to the user terminal 2B. Based on these data transmitted from the user terminal 2A, the user terminal 2B performs a process of determining whether or not the received captured image data has been tampered with (tampering determination process). The tampering determination process will be explained later.

[0032] The network 3 may be, for example, the Internet, a home network, a LAN (Local Area Network), a satellite communication network, or any other type of network. The server device 4 is an information processing device that manages the video chat system, and has a function of managing personal information of users, a function of storing records and histories of calls between users, and the like.

[0033] <1-2. User terminal configuration> FIG. 2 is a block diagram showing an example of the internal configuration of the user terminal 2. The user terminal 2 includes an array sensor 10, an image processor 11, a control unit 12, a memory unit 13, an input unit 14, a communication unit 15, and an output unit 16 in order to execute the functions of the user terminal 2A.

[0034] The array sensor 10 includes a pixel array 21, a hash value generation unit 22, and an encryption processing unit . The pixel array 21 is composed of multiple pixels arranged in one or two dimensions, each of which has a light receiving element for visible light or invisible light. Each pixel receives light that is incident through a camera optical system (not shown) provided in the user terminal 2, and performs photoelectric conversion to obtain captured image data.

[0035] The hash value generation unit 22 generates (calculates) a hash value from the captured image data output from the pixel array 21. The generated hash value is output to an encryption processing unit 23 at a downstream stage.

[0036] The encryption processing unit 23 encrypts the input hash value. A private key is used for the encryption processing. That is, the encryption processing unit 23 generates a private key as a signature generation key and a public key as a signature verification key.

[0037] The array sensor 10 outputs the encrypted hash value and the generated public key to the control unit 12. In addition, the array sensor 10 outputs the captured image data generated in the pixel array 21 to the image processor 11.

[0038] Although not shown in the drawings, the array sensor 10 has, as hardware, components such as an image sensor device, a memory device such as a DRAM (Dynamic Random Access Memory), and a processor. These components are arranged flat or in a stacked structure to form an integrated device. The array sensor 10 of this embodiment is equipped with a function for generating a hash value and a function for performing encryption, and is a device that can be called an intelligent array sensor.

[0039] The image processor 11 performs a process of digitizing the photoelectric conversion signal. Furthermore, the image processor 11 performs a process of temporarily storing the digitized signal in a buffer, and various kinds of signal processing (image processing) required for reading the signal stored in the buffer.

[0040] Assumed various types of signal processing include image quality adjustment processing such as color correction, gamma correction, color gradation processing, gain processing, edge enhancement processing, etc. It is also assumed that the image processor 11 performs processing to change data size, such as data compression processing, resolution conversion, and frame rate conversion. Parameters used for each process are set in the image processor 11. For example, these include setting values ​​for color and brightness correction coefficients, gain values, compression ratios, frame rates, etc. The image processor 11 performs the necessary processing using the parameters set for each process.

[0041] The control unit 12 is configured, for example, with a microcomputer equipped with a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory), and performs overall control of the user terminal 2 by the CPU executing various processes in accordance with programs stored in the ROM or programs loaded into the RAM (Random Access Memory). For example, the control unit 12 issues instructions to the array sensor 10 to control the execution of various processes such as the acquisition process of captured image data. The control unit 12 also controls the image processor 11 to execute various processes.

[0042] The control unit 12 also controls writing and reading of various data to and from the memory unit 13. The memory unit 13 is a non-volatile storage device such as a hard disk drive (HDD) or a flash memory device, and is used as a storage destination (recording destination) for various data such as captured image data obtained by the array sensor 10.

[0043] Furthermore, the control unit 12 acquires an input signal from the input unit 14 and performs control according to the input signal. The input unit 14 includes, for example, a keyboard, a mouse, or a touch panel used by a user who uses video chat. Furthermore, a speed sensor, an acceleration sensor, or an angular velocity sensor provided in a smartphone or a camera is also an example of the input unit 14. The control unit 12 acquires the operation information and sensor information, and executes various processes.

[0044] Furthermore, the control unit 12 performs various data communications with external devices via the communication unit 15. The communication unit 15 is configured to be capable of performing data communications with external devices via the network 3 shown in FIG. In response to a request from another user terminal 2B shown in Figure 1, the control unit 12 is capable of transmitting various data, such as image data obtained by the array sensor 10 and stored in the memory unit 13, to the user terminal 2B via the communication unit 15.

[0045] Furthermore, the control unit 12 outputs an output signal to the output unit 16. The output unit 16 is configured with, for example, a display such as an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) panel, a speaker, and the like.

[0046] The control unit 12 has various functions related to electronic signatures. Specifically, the control unit 12 has an acquisition unit 31, a decryption processing unit 32, a hash value calculation unit 33, a hash value comparison unit 34, a tampering determination unit 35, a biometric analysis processing unit 36, a biometric determination unit 37, an individual analysis processing unit 38, an individual determination unit 39, an output control unit 40, and a state determination unit 41. These units are functions of the user terminal 2B, that is, units for realizing the function of determining whether the captured image data received from the user terminal 2A has been tampered with. In this embodiment and other embodiments described below, the user terminal 2 does not need to have all the functions, and may have a configuration including only some of the above-mentioned units.

[0047] The acquisition unit 31 executes a process of acquiring the captured image data and the encrypted hash value generated by the user terminal 2 A. The acquisition unit 31 also acquires a public key for decrypting the encrypted hash value.

[0048] The decryption processing unit 32 performs a process of decrypting the encrypted hash value. A public key is used for the decryption.

[0049] The hash value calculation unit 33 performs a process of calculating a hash value from the received captured image data.

[0050] The hash value comparison unit 34 executes a process of comparing the hash value decrypted by the decryption processing unit 32 with the hash value calculated by the hash value calculation unit 33 .

[0051] The tampering determination unit 35 executes a tampering determination process to determine whether the captured image data has been tampered with or not based on the comparison result of the hash values. As the determination result of the tampering determination, there are a determination result that can guarantee that the captured image data has not been tampered with (non-tampering determination) and a determination result that cannot guarantee that the captured image data has not been tampered with (tampering determination). In other words, even if the captured image data has not been tampered with, it is not necessarily determined that the captured image data has not been tampered with. If it is not guaranteed that the captured image data has not been tampered with, the captured image data is determined to have been tampered with even if the captured image data has not actually been tampered with.

[0052] The biometric analysis processing unit 36 ​​performs processing to analyze the received captured image data and extract biometric information. For example, the biometric analysis processing unit 36 ​​extracts biometric information by comparatively analyzing a plurality of received captured image data.

[0053] The biometric determination unit 37 performs a process of determining whether or not the subject is a living body based on the analysis result of the biometric analysis processing unit 36. For example, it determines whether or not the captured image data is obtained by capturing an image of a living person, that is, whether or not the subject is a living human. It may also determine how many people are captured as subjects.

[0054] The individual analysis processing unit 38 performs processing to analyze the received captured image data and extract individual features of the subject. Individual features are information that can identify an individual, such as skin color, eye position, ratio of eyebrow length to face width, nose position and shape, lip position, shape, form and color, and other mole position and number. In addition, hairstyle and color are not universal, but may be extracted as individual features. Furthermore, when the subject's entire body is captured, the ratio, length and size of each part, such as shoulder width and limb length, may be extracted. Note that features for identifying individual organisms other than humans, such as dogs and cats, may also be extracted as individual features.

[0055] The individual determination unit 39 performs a process of determining an individual based on the analysis result of the individual analysis processing unit 38. For example, this process may identify the individual by transmitting the analysis result to another information processing device, or the individual may be identified in the user terminal 2B. As an example of requesting the other information processing device to identify an individual, for example, the individual feature amount for each user is stored in the server device 4, and the individual feature amount extracted is transmitted to the server device 4 to request the individual to be identified. In this case, the server device 4 performs a process of comparing the stored individual feature amount with the received extracted individual feature amount, and the individual is identified.

[0056] The output control unit 40 performs output control for notifying the user according to the judgment result of the tampering judgment unit 35, the judgment result of the liveness judgment unit 37, and the judgment result of the individual judgment unit 39. For example, the output control unit 40 performs user interface control for controlling display on the display unit. The output control unit 40 may also perform output control of sound, etc.

[0057] The state determination unit 41 performs a process of determining the state (guaranteed state) of the captured image data according to the results of various determination processes on the captured image data received from the user terminal 2A. The states of the captured image data include, for example, a non-secure state ST0 in which it is not guaranteed that the captured image data has not been tampered with, a first secured state ST1 in which it is guaranteed that the captured image data has not been tampered with, a second secured state ST2 in which it is guaranteed that the captured image data has not been tampered with and the subject is determined to be a living organism, and a third secured state ST3 in which it is guaranteed that the captured image data has not been tampered with, the subject is determined to be a living organism, and the subject is recognized as an individual.

[0058] 2, an example in which the array sensor 10 and the control unit 12 are provided separately has been described, but the control unit 12 may be provided as a part of the array sensor 10. This also applies to other embodiments described later.

[0059] <1-3. Overall flow> An example of the overall flow when a user A using a user terminal 2A makes a call request by video chat to another user B using a user terminal 2B is shown in Figs. 3, 4 and 5.

[0060] First, when user A starts a program for using the video chat system (hereinafter, referred to as the "video chat program"), user terminal 2A starts the camera function of user terminal 2A according to the application program and performs an image capture operation (see FIG. 3A). At this time, user terminal 2A generates a hash value from the captured image data obtained by photoelectric conversion and performs encryption processing.

[0061] Next, when user A operates the video chat program to select a call partner and start making a call, an incoming call screen 50 indicating that a call is coming from user A is displayed on the display unit of user terminal 2B. The example of incoming call screen 50 shown in Fig. 4 is an example in which user terminal 2B is a mobile terminal such as a smartphone.

[0062] As shown in FIG. 4, the incoming call screen 50 has a caller display section 61 that displays an icon image of the caller and the registered name of the caller, user A, a guarantee icon 62 that indicates that the captured image data received from the user terminal 2A has not been tampered with, an explanation 63 that explains in text that the captured image data has not been tampered with, a call start button 64, and a call reject button 65 arranged thereon.

[0063] The caller display field 61 may display an icon image registered by the caller, user A, or may display an image based on captured image data captured by the user terminal 2A at the time of the current call request.

[0064] For example, an icon image corresponding to the above-mentioned state (ST0 to ST3) of the captured image data may be displayed as the guarantee icon 62. In the example shown in Fig. 4, a first guarantee icon MK1 corresponding to a first guarantee state ST1 in which it is guaranteed that the captured image data currently received from the user terminal 2A has not been tampered with is displayed.

[0065] The state in which the first guarantee icon MK1 is displayed on the guarantee icon 62 may be a state in which it is guaranteed that the captured image data has not been tampered with because the user terminal 2B has performed the tampering determination process but has not performed any other determination processes (biometric determination process and individual determination process), or it may be a state in which it is guaranteed that the captured image data has not been tampered with because the user terminal 2B has performed the tampering determination process as well as the biometric determination process and individual determination process but has not been able to determine that the subject is a living body or has not been able to individually recognize the subject.

[0066] By visually checking the guarantee icon 62, user B can confirm that the captured image data received from the caller before starting the call has not been tampered with, i.e., that the image that has been sent is actually captured by the user terminal 2A, and that illegally obtained image data has not been misused to make a call request by someone posing as the caller. This allows User B to distinguish fraudulent call requests and prevent damage from occurring.

[0067] When user B operates the call start button 64, a call screen 51 is displayed on the display unit of the user terminal 2B. An example of the call screen 51 is shown in FIG.

[0068] On the call screen 51, a partner image display field 66, a self-portrait display field 67, and various function icons 68 are displayed. In the other party image display field 66, moving image data captured by the user terminal 2A used by the other party is displayed. In the self-portrait display field 67, the moving image data of the user (that is, user B) displayed on the user terminal 2A of the other party is displayed in a reduced size.

[0069] A guarantee icon 62 is superimposed on the other party image display field 66. In the example shown in Fig. 5, a first guarantee icon MK1, which indicates a state in which it is only guaranteed that the captured image data has not been tampered with, is superimposed as the guarantee icon 62.

[0070] The function icons 68 are provided on the screen as operators for executing various functions. The various functions include, for example, a function for ending a call, a function for turning off one's own audio, a function for pausing a call, a function for changing the volume, a function for stopping transmission of one's own video image, and a function for chatting by text. These functions are merely examples, and function icons 68 for executing other functions may be displayed on the call screen 51. In addition, some of the above-mentioned function icons 68 may not be displayed. Furthermore, the user may be able to set the display and non-display states of these icons.

[0071] Since the guarantee icon 62 continues to be displayed on the call screen 51, user B can confirm that the video data sent by the other party (user A) is actually an image captured by the user terminal 2A, and can recognize if the call is hijacked by an unauthorized person during the call.

[0072] To achieve this, it is desirable that the user terminal 2A performs a hash value generation process, an encryption process, and a transmission process for the captured image data at regular intervals, such as every few seconds, and that the user terminal 2B accordingly performs a decryption process, a comparison process, and a tampering determination process for the received hash value.

[0073] Although an example in which the first guarantee icon MK1 is displayed as the guarantee icon 62 has been given in FIG. 4 and FIG. 5, other icon images may be displayed depending on the guarantee state of the received captured image data.

[0074] For example, in the second guaranteed state ST2 in which it is guaranteed that the captured image data has not been tampered with and the subject is determined to be a living body, the second guarantee icon MK2 shown in Fig. 6 may be displayed as the guarantee icon 62 on each screen. Also, the explanation 63 shown in Fig. 6 is displayed instead of the explanation 63 on the incoming call screen 50 in Fig. 5.

[0075] In addition, in the third guarantee state ST3 in which it is guaranteed that the captured image data has not been tampered with, the subject is determined to be a living organism, and the individual recognition of the subject is performed, the third guarantee icon MK3 shown in Fig. 7 may be displayed as the guarantee icon 62 on each screen. Moreover, the explanation 63 shown in Fig. 7 is displayed instead of the explanation 63 on the incoming call screen 50 in Fig. 5.

[0076] Furthermore, in the case of a non-secured state ST0 in which it is not guaranteed that the captured image data has not been tampered with, a non-guaranteed icon MK0 shown in Fig. 8 may be displayed as the guarantee icon 62 on each screen. Also, instead of the explanation 63 on the incoming call screen 50 in Fig. 5, the explanation 63 shown in Fig. 8 is displayed.

[0077] Note that each icon image is merely an example, and it is sufficient that each icon image is configured so that the guarantee state of the received captured image data can be identified.

[0078] <1-4. Processing on the sending side> An example of the process executed by the user terminal 2A on the sending side to realize the above-mentioned video chat system is shown in Fig. 9. Note that the series of processes shown in Fig. 9 are processes executed when, for example, user A operates the user terminal 2A to start up a video chat program.

[0079] First, in step S101, the control unit 12 of the user terminal 2A activates the camera function and performs an imaging operation using the array sensor.

[0080] Next, in step S102, the control unit 12 of the user terminal 2A executes a process of generating a hash value from captured image data obtained by the imaging operation.

[0081] Next, in step S103, the control unit 12 of the user terminal 2A performs encryption processing on the generated hash value using a private key.

[0082] When user A selects a call partner and performs an operation to transmit a call start request, the control unit 12 of the user terminal 2A performs a call start request transmission process in step S104. The call start request is transmitted to the user terminal 2B used by user B via the server device 4, for example. In the call start request transmission process in step S104, the captured image data acquired in step S101, the encrypted hash value in step S103, and the generated public key are transmitted to the user terminal 2B.

[0083] <1-5. Processing on the receiving side> FIG. 10 shows an example of the process executed by the user terminal 2B on the receiving side to realize the above-mentioned video chat system.

[0084] When the user terminal 2B receives the captured image data, the encrypted hash value, and the public key and starts up the video chat program, the control unit 12 of the user terminal 2B performs an acquisition process for each piece of received data in step S201.

[0085] Next, in step S202, the control unit 12 of the user terminal 2B executes a process of decrypting the encrypted hash value. Furthermore, in step S203, the control unit 12 of the user terminal 2B executes a process of calculating a hash value from the received captured image.

[0086] In step S204, the control unit 12 of the user terminal 2B performs a comparison process between the calculated hash value and the decrypted hash value.

[0087] The control unit 12 of the user terminal 2B performs a tampering determination process in step S205. This process is a process for determining whether or not tampering has occurred based on the comparison result of the hash values. Note that regardless of whether or not tampering has actually occurred, if there is a possibility of tampering, it is determined that there is no guarantee that the data has not been tampered with. Specifically, if the two hash values ​​match, the determination result is a non-tampering determination, whereas if the two hash values ​​do not match, the determination result is a tampering determination.

[0088] In step S206, the control unit 12 of the user terminal 2B executes a branching process according to the determination result of the tampering determination process. When it can be guaranteed that the data has not been tampered with (non-tampering determination), the control unit 12 of the user terminal 2B performs output control based on the first guarantee state ST1 in step S207. The output control based on the first guarantee status ST1 is, for example, a process of displaying the first guarantee icon MK1 and the explanatory text 63 on the incoming call screen 50 of FIG.

[0089] Another aspect of the output control may be, for example, a process of performing audio output according to the first guarantee state ST1.

[0090] On the other hand, if it cannot be guaranteed that the data has not been tampered with (tampering determination), that is, if the data is in the non-secured state ST0, the control unit 12 of the user terminal 2B performs output control based on the non-secured state ST0 in step S208. The output control based on the non-secured state ST0 is, for example, a process of displaying a non-guaranteed icon MK0 and an explanatory text 63 (see FIG. 8) on the incoming call screen 50 in FIG. 4. In addition, audio output, etc. may be executed in this process.

[0091] As described above, when guaranteeing that the data has not been tampered with at regular intervals, the user terminal 2A and the user terminal 2B execute the processes in Fig. 9 and Fig. 10 at regular intervals. This allows the user B to recognize if a takeover by an unauthorized person occurs during a call.

[0092] <2. Second embodiment> In the second embodiment, in order to increase the secrecy of the call, the captured image data transmitted from the user terminal 2A to the user terminal 2B is encrypted. In the following description, the same parts as those in the first embodiment are given the same reference numerals and the description thereof will be omitted as appropriate.

[0093] <2-1. System configuration> In the user terminal 2 (2A, 2B) of the information processing system 1 in the second embodiment, the array sensor 10 includes an image encryption processing unit . Therefore, the configuration of the image encryption processing unit 24 will be mainly described here.

[0094] The image encryption processing unit 24 encrypts the captured image data output from the pixel array 21. There are various possible examples of the encryption process, but here, an example of encryption process using photoelectric random numbers will be described.

[0095] The photoelectric random number used by the image encryption processing unit 24 when encrypting the captured image data that is the target data for the electronic signature means a random number obtained based on the photoelectric conversion by the array sensor 10 . Specifically, in this example, the value of the electrical signal for each pixel obtained by photoelectric conversion of the array sensor 10 is obtained as a photoelectric random number, and an encryption key is generated.

[0096] FIG. 12 shows an example of a method for generating an encryption filter (encryption key) for image data based on a photoelectric random number. First, the left side of the figure illustrates the value of an electrical signal for each pixel obtained by photoelectric conversion of the array sensor 10. In this example, the photoelectric random number is the pixel value (brightness value) of an image (still image) obtained by imaging with the array sensor 10. In the following description, the frame image captured to obtain photoelectric random numbers, in other words, the frame image that is the source of the photoelectric random numbers, is referred to as a "seed frame".

[0097] In this example, the value of such an electrical signal for each pixel is not used as a photoelectric random number, but rather, as shown in the example on the right side of the figure, a photoelectric random number is generated in a format in which at least a part of the value of the electrical signal for each pixel is assigned to a pixel position different from the pixel position at which the value of the electrical signal was obtained. In other words, the pixel positions of the electrical signal value for each pixel are shuffled to generate a photoelectric random number. Then, in this example, the photoelectric random number generated in this way is used as an encryption key (encryption filter) for the captured image data. By using photoelectric random numbers in a format in which pixel positions are shuffled as described above, it is made more difficult to decipher the encryption key, thereby improving security, compared to using photoelectric random numbers in which the electrical signal values ​​for each pixel are directly assigned to the pixel positions from which those electrical signal values ​​were obtained.

[0098] Here, when generating the encryption key, the value of the electrical signal for each pixel can be modulated using a predetermined algorithm. For example, the value obtained by multiplying the value of the electrical signal for each pixel by a predetermined coefficient can be assigned to the pixel as a photoelectric random number. Alternatively, when the value of the electrical signal for each pixel includes a value after the decimal point, a method can be adopted in which the value of the last few decimal places is converted to an integer to be used as the photoelectric random number. In generating the encryption key, it is not essential to shuffle the pixel positions as described above, and the value of the electrical signal for each pixel can itself be used as the encryption key.

[0099] Here, conventionally, pseudo-random numbers generated by software have often been used as random numbers for encryption. However, because pseudo-random numbers are generated by an algorithm that calculates numerical values, and true random numbers cannot be generated, there is a risk that the encryption key will be decrypted and copied. In contrast, the above-mentioned photoelectric random numbers can be true random numbers, and by generating an encryption key based on the photoelectric random numbers, it is possible to make it difficult to decrypt the encryption key.

[0100] Photoelectric random numbers may be used for the encryption of the above-mentioned hash value. Specifically, a private key used for encryption may be generated based on the photoelectric random numbers. This makes it difficult to decrypt the private key, thereby improving security.

[0101] <2-2. Signals to be encrypted> Conventionally, when encrypting an image signal obtained by imaging with the array sensor 10, it has been common practice to temporarily store the image signal read out from the array sensor 10 in memory in plain text form, and then encrypt the stored image signal. However, when this type of encryption method is used, it becomes possible for an attacker to use malware or other means to intentionally cause an error during encryption, outputting the memory contents as a dump file and then copying the plaintext stored in the memory.

[0102] Therefore, in this embodiment, the signals read from the pixels of the array sensor 10 are encrypted so that image signals in plain text are not stored in the memory. Specifically, in this example, the image encryption processing unit 24 shown in FIG. 11 performs amplitude control on the read signal from the pixels of the array sensor 10 using a coefficient corresponding to the encryption key (encryption filter) shown in FIG. 12, thereby encrypting the read signal.

[0103] FIG. 13 is a diagram showing an image of encryption of a read signal by the image encryption processing unit 24. As shown in FIG. As shown in the figure, a read signal (charge signal in this case) from each pixel in pixel array 21 of array sensor 10 is multiplied by a coefficient corresponding to an encryption key by an amplifier provided in image encryption processing unit 24. In image encryption processing unit 24 of user terminal 2 shown in Fig. 11, amplitude control is performed on the read signal for each pixel at the analog signal stage in this manner, and then image data on a frame-by-frame basis is temporarily stored in a buffer. The image data temporarily stored in the buffer is read out at an appropriate timing and provided to image processor 11, etc.

[0104] That is, the image encryption processing unit 24 sets a coefficient corresponding to the encryption key in the amplifier, so that the readout signal from each pixel in the array sensor 10 is encrypted.

[0105] 13 is merely an image diagram, and it is not essential that an amplifier be provided for each pixel in the image encryption processing unit 24. For example, when batch reading is performed as in a CCD (Charged-coupled devices) image sensor, the image encryption processing unit 24 may have one amplifier common to each pixel. In that case, the amplitude control for each pixel is performed by time division.

[0106] In the above, an example of encrypting a read signal in the form of an analog signal has been given as an example of encrypting a read signal, but it is also possible to encrypt a read signal in the form of a digital signal after A / D conversion.

[0107] Since it is extremely difficult to illegally obtain an analog signal from outside, security can be improved by employing a configuration in which the analog read signal is encrypted.

[0108] When encrypting an analog read signal, there is a concern that the reproducibility of an image obtained by decrypting the encrypted image may decrease. However, for example, when the image is used to analyze whether or not a subject captured in the image is a living organism, the reproducibility of the image needs only to be sufficient to withstand biological analysis processing, and it is considered that no practical problem will arise. The same applies to the case where individual analysis processing of the subject captured in the image is performed.

[0109] On the other hand, when the digital read signal is encrypted, the accuracy of the encryption process is improved, and the reproducibility of the image can be improved.

[0110] The encryption performed on the read signal as described above is a type of encryption using a stream cipher system, which encrypts plaintext in predetermined data units such as bits or bytes. In the stream encryption method, it is not necessary to make the data length of the signal to be encrypted uniform, and therefore pre-processing of the signal is not required. Therefore, by adopting the stream encryption method, it is possible to speed up the encryption process.

[0111] Here, even if encryption is applied to an analog read signal, the result is still encrypted captured image data that is obtained as the target data for the electronic signature. From this point of view, in this specification, encryption at the analog signal stage is considered to be included in the category of encryption of captured image data.

[0112] <2-3. Processing on the sending side> An example of processing executed by the transmitting user terminal 2A in the second embodiment is shown in Fig. 14. Note that processing similar to the processing on the transmitting side in the first embodiment shown in Fig. 9 is given the same reference numerals and description thereof will be omitted as appropriate.

[0113] In step S101, the control unit 12 of the user terminal 2A performs an imaging operation using the array sensor by using the camera function.

[0114] In step S111, the user terminal 2A encrypts the captured image data. The encryption process of the captured image data is executed by the image encryption processing unit 24 of the array sensor 10.

[0115] The encryption process for captured image data will be specifically described with reference to Fig. 15. Note that at least a part of the process described below can also be realized as a hardware process.

[0116] In step S301, the image encryption processing unit 24 acquires captured image data generated by capturing a still image. This captured image data is still image data that is the source of generating an encryption key, and is obtained by the array sensor 10 capturing an image for one frame (reading out the charge for each pixel) in response to an instruction from the control unit 12. By acquiring still image data in step S301, image data is stored as a seed frame in a memory or the like provided in the array sensor 10.

[0117] In step S302 following step S301, the image encryption processing unit 24 executes a pixel value uniformity check process. This uniformity check process is a process for checking the uniformity of the luminance value for each pixel in the seed frame, and specifically, the image encryption processing unit 24 counts the number of pixels whose luminance value is zero or the number of pixels whose luminance value is a saturated value (maximum value). The process of checking the uniformity of pixel values ​​can also be executed as a process of checking the uniformity of read signal values.

[0118] In step S303 following step S302, the image encryption processing unit 24 determines whether or not the uniformity is excessive. Specifically, it determines whether or not the number of pixels counted in step S302 is equal to or greater than a predetermined threshold value (for example, a value corresponding to 30% to 50% of the number of valid pixels). If the number of pixels counted in step S302 is greater than or equal to the threshold value and the result is that the uniformity is excessive, the image encryption processing unit 24 proceeds to step S304 to execute a process of erasing the seed frame, i.e., a process of erasing the image data as the seed frame stored in a memory, etc., and then returns to step S301. This allows the seed frame to be re-imaged in response to a case where the randomness of the pixel values ​​of the seed frame is low. That is, the photoelectric random numbers can be re-acquired in response to a case where the randomness of the photoelectric random numbers is low. Therefore, it is possible to prevent encryption from being performed using an encryption key based on a random number with low randomness, thereby improving security.

[0119] On the other hand, if it is determined in step S303 that the number of counted pixels is not equal to or greater than the threshold value and the uniformity is not excessive, the image encryption processing unit 24 proceeds to step S305 to generate an encryption key. Specifically, in this example, an encryption key representing a coefficient to be set for each amplifier in the image encryption processing unit 24 is generated based on the luminance value of each pixel in the seed frame.

[0120] Here, in this example, in the processing of step S305, an encryption key is not generated based on a photoelectric random number in a format in which the luminance value for each pixel is directly assigned to the pixel position from which the luminance value was obtained, but rather, an encryption key is generated based on a photoelectric random number in a format in which at least a portion of the luminance value for each pixel is assigned to a pixel position other than the pixel position from which the luminance value was obtained. This makes it difficult to decrypt the encryption key, thereby improving security.

[0121] In step S306 following step S305, the image encryption processing unit 24 executes a process of deleting the seed frame, that is, a process of deleting the image data as the seed frame stored in the memory or the like in the acquisition process in step S301. By performing this seed frame erasure process, it is possible to prevent the image on which the photoelectric random numbers are based from leaking out and the photoelectric random numbers from being estimated.

[0122] For example, when the processing capacity of the image encryption processing unit 24 is high or the image size of the seed frame is small, it is not essential to temporarily store the seed frame in a memory, etc. In this case, the deletion process of step S306 and the deletion process of step S304 are not required.

[0123] In the next step S307, the image encryption processing unit 24 deletes the existing key, if there is one. For example, if the process shown in Fig. 15 is started at regular time intervals, the image encryption processing unit 24 holds an encryption key due to the process of step S308 that was previously performed. The process of step S307 is a process of deleting the existing encryption key thus held. By performing this process of erasing the existing key, it is possible to prevent the leakage of encryption keys used for encryption in the past, and to prevent signals that were encrypted in the past from being illegally decrypted.

[0124] In the next step S308, the image encryption processing unit 24 executes an encryption key storage process, that is, a process of storing in the memory 16 the encryption key generated in step S305. The processes in steps S307 and S308 may be performed simultaneously by overwriting the existing encryption key with a new encryption key. In response to the execution of the storage process in step S308, the image encryption processing unit 24 ends the series of processes shown in FIG.

[0125] In the user terminal 2A, the encryption key stored in step S308 is used to encrypt image data (captured image data) obtained by imaging with the array sensor 10. Specifically, after the process shown in Fig. 15 is completed, the image encryption processing unit 24 sets a coefficient for each pixel based on the stored encryption key in each amplifier, so that the image signal obtained by imaging with the array sensor 10 is encrypted based on the stored encryption key. In this example, the array sensor 10 is assumed to capture a moving image, and the encryption by the image encryption processing unit 24 is performed on each frame image constituting the moving image.

[0126] In this example, the captured image data as a moving image encrypted as described above is stored in the memory unit 13 under the control of the control unit 12. The control unit 12 is capable of transmitting the captured image data thus stored in the memory unit 13 to the user terminal 2B via the communication unit 15.

[0127] As can be understood from the above description, in this example, the encryption of image data is performed based on a photoelectric random number obtained in a frame period different from that of the image data to be encrypted. This makes it more difficult to deduce the encryption key from the encrypted image, thereby improving security. It is also possible to encrypt image data based on a photoelectric random number obtained during the same frame period as the image data to be encrypted.

[0128] The image encryption processing unit 24 can also execute the process shown in FIG. 15 when it detects unauthorized access. 15 is started in response to detection of unauthorized access, so that acquisition of a photoelectric random number (step S301) and generation of an encryption key (step S305) are executed in response to detection of unauthorized access. That is, a photoelectric random number is reacquired in response to detection of unauthorized access, and an encryption key is regenerated based on the reacquired photoelectric random number. This provides tamper resistance on the software side.

[0129] Returning to the explanation of Figure 14. The control unit 12 of the user terminal 2A generates a hash value in step S102, encrypts the hash value using a private key in step S103, and transmits a call start request to the user terminal 2B (or the server device 4) in step S104. As a result, the captured image data, the encrypted hash value, and the public key are transmitted from the user terminal 2A to the user terminal 2B.

[0130] <2-4. Processing on the receiving side> FIG. 16 shows an example of processing executed by the receiving user terminal 2B in the second embodiment. Note that the same processes as those on the receiving side in the first embodiment shown in FIG. 10 are denoted by the same reference numerals and the description thereof will be omitted as appropriate.

[0131] When the data is received from the user terminal 2A and a video chat program is started in the user terminal 2B, the control unit 12 of the user terminal 2B acquires the encrypted captured image data, the encrypted hash value, and the public key in step S201A.

[0132] Next, in step S202, the control unit 12 of the user terminal 2B executes a process of decrypting the encrypted hash value.

[0133] Next, in step S203A, the control unit 12 of the user terminal 2B calculates a hash value from the captured image in an encrypted state.

[0134] The control unit 12 of the user terminal 2B compares the calculated hash value with the decrypted hash value in step S204, performs tampering determination processing in step S205, and performs branching processing according to the tampering determination result in step S206.

[0135] If the control unit 12 of the user terminal 2B determines that it can guarantee that the data has not been tampered with, it performs output control based on the first secured state ST1 in step S207, and if it determines that it cannot be guaranteed that the data has not been tampered with, it performs output control based on the non-secured state ST0 in step S208.

[0136] As a result, the first guarantee icon MK1 and the non-guaranteed icon MK0 are displayed on the incoming call screen 50 presented to the user B.

[0137] <3. Third embodiment> In the information processing system 1 in the third embodiment, a display is performed according to the security status of the captured image data received at the receiving user terminal 2B. Specifically, a screen display is performed according to the determination result (non-secured status ST0, first secured status ST1, second secured status ST2, third secured status ST3) of the status determination unit 41 for the captured image data received from the user terminal 2A.

[0138] The system configuration and the configuration of the user terminal 2 are the same as those described in the first embodiment, and therefore the description will be omitted. Therefore, hereinafter, the flow of processing in the user terminal 2A on the transmitting side and the flow of processing in the user terminal 2B on the receiving side will be described.

[0139] <3-1. Processing on the sending side> The user terminal 2A on the transmitting side executes, for example, each process shown in FIG. 9. However, in the third embodiment, a biometric analysis is performed. The biometric analysis is a process of extracting biometric information from a plurality of still images. Therefore, the imaging operation in step S101 is performed to obtain a plurality of still images. In the hash value generation process in step S102, a hash value is generated for each of the multiple still image data. Furthermore, in the encryption process in step S103, each of the multiple generated hash values ​​is encrypted.

[0140] In the call start request transmission process in step S104, a plurality of captured image data and their corresponding encrypted hash values ​​and public keys are transmitted to the user terminal 2B.

[0141] <3-2. Processing on the receiving side> In the third embodiment, the user terminal 2B on the receiving side executes, for example, each process shown in FIGS. 17 and 18 upon receiving the captured image data, the encrypted hash value, and the public key from the user terminal 2A. Note that the same processes as those shown in FIG. 10 are denoted by the same reference numerals and the description thereof will be omitted as appropriate.

[0142] The control unit 12 of the user terminal 2B executes each process from step S201 to step S205 to determine whether or not it can be guaranteed that each received captured image data has not been tampered with.

[0143] In step S206, the control unit 12 of the user terminal 2B performs branching processing according to the result of the tampering determination processing. Specifically, if it cannot be guaranteed that the message has not been tampered with, the control unit 12 of the user terminal 2B performs output control based on the non-secured state ST0 in step S208. As a result, a non-secured icon MK0 and an explanation 63 as shown in FIG. 8 are displayed on the incoming call screen 50 presented to the user in the user terminal 2B.

[0144] On the other hand, if it is determined that it can be guaranteed that the data has not been tampered with, the control unit 12 of the user terminal 2B executes each process shown in FIG. Specifically, the control unit 12 of the user terminal 2B executes a biometric analysis process in step S211. The biometric analysis process is a process for extracting biometric information from a plurality of still images, and is a process for extracting information unique to a living body.

[0145] For example, nystagmus can be detected from a living human being. Although nystagmus cannot be extracted from a single piece of captured image data, it can be detected by extracting differential information around the eyeballs from multiple captured image data.

[0146] Alternatively, the heartbeat can be detected from a living human being. The color of human skin changes slightly in accordance with the heartbeat. Such changes can be extracted from multiple captured image data.

[0147] Furthermore, pupil response and lip sync caused by brightness can also be extracted from multiple captured image data.

[0148] In the biological analysis process in step S211, such biological information is extracted.

[0149] Next, the control unit 12 of the user terminal 2B executes a living body determination process in step S212. This process is a process for determining whether or not the subject is a living body based on the extracted living body information. For example, when nystagmus is detected as living body information from multiple captured image data, the subject is determined to be a living body.

[0150] In step S213, the control unit 12 of the user terminal 2B performs branching processing based on the result of the determination as to whether or not the subject is a living body. If the subject is not determined to be a living organism (or is determined to be a non-living organism), it is determined that the captured image data has not been tampered with, but the subject cannot be guaranteed to be a living organism. In this case, the control unit 12 of the user terminal 2B performs output control based on the first guarantee state ST1 in step S207. As a result, the first guarantee icon MK1 and explanation 63 as shown in FIG. 4 are displayed on the incoming call screen 50 presented to the user on the user terminal 2B.

[0151] On the other hand, if it is determined that the subject is a living organism, the control unit 12 of the user terminal 2B performs an individual analysis process in step S214. The individual analysis process is a process for extracting individual information for identifying an individual from the received captured image data.

[0152] After completing the individual analysis process, the control unit 12 of the user terminal 2B performs an individual determination process in step S215. The individual determination process is a process for determining an individual based on the extracted individual information. The individual determination process may identify the subject as an individual, or may determine whether or not information capable of identifying an individual has been extracted.

[0153] In step S216, the control unit 12 of the user terminal 2B executes a branching process based on the determination result of the individual determination process. If the individual cannot be identified, or if information that can identify the individual cannot be extracted, it is a case where it is possible to guarantee that the captured image data has not been tampered with and that the subject is a living organism, but the individual cannot be identified. In this case, the control unit 12 of the user terminal 2B performs output control based on the second guarantee state ST2 in step S217. As a result, the second guarantee icon MK2 and an explanation 63 as shown in FIG. 6 are displayed on the incoming call screen 50 presented to the user on the user terminal 2B.

[0154] On the other hand, if an individual can be identified, or if information capable of identifying an individual can be extracted, the control unit 12 of the user terminal 2B performs output control based on the third guarantee state ST3 in step S218. As a result, the third guarantee icon MK3 and explanation 63 as shown in FIG. 7 are displayed on the incoming call screen 50 presented to the user on the user terminal 2B.

[0155] As described above, the user terminal 2B displays the guarantee icon 62 and the explanatory text 63 on the incoming call screen 50 depending on the presence or absence of the possibility of tampering with the received captured image data and the information that can be extracted. This allows the user B to appropriately decide whether or not to answer the incoming call, and prevents the user B from becoming a victim of fraud or the like.

[0156] The captured image data used in the tampering determination process, biometric analysis process, and individual analysis process may be the sender's video data actually presented to user B. This eliminates the need to obtain dedicated captured image data for use in the tampering determination process and each analysis process, and reduces the processing load and communication capacity.

[0157] <3-3. Application Examples> The configurations described in the first, second and third embodiments can also be applied to a text chat system that does not involve voice. A specific description will be given with reference to FIG. 19, which shows a text chat screen 52 presented to a user who uses the text chat system.

[0158] The text chat screen 52 is arranged with participant icons 70 set for each user participating in the chat, guarantee icons 71 indicating the guarantee status of each participant, speech bubbles 72 displaying the content of comments, an input field 73 for inputting the content of comments, and function icons 74 for executing various functions.

[0159] The participant icons 70 include participant icons 70A for participants other than the participant himself (user B) and a participant icon 70B for the participant himself. The guarantee icon 71 also includes a guarantee icon 71A for participants other than the participant himself (user B) and a guarantee icon 71B for the participant himself.

[0160] In the example of FIG. 19, the guarantee icon 71A for the other user is the second guarantee icon MK2, and the guarantee icon 71B for the user himself is the third guarantee icon MK3. In this way, a guarantee icon 71 is displayed according to the guarantee status of each participant.

[0161] Note that the guarantee icon 71B for the user himself / herself only needs to be displayed on at least the text chat screen 52 presented to the other participants. In other words, the guarantee icon 71 for user B himself / herself does not need to be displayed on the text chat screen 52 presented to user B. However, by displaying his / her own guarantee icon 71, it is possible for the user to understand how his / her guarantee status is displayed to the other participants. This makes it possible to perform operations such as re-executing the various determination processes and analysis processes described above when the intended guarantee status is not achieved.

[0162] Unlike the above-mentioned video chat system, a text chat system does not originally require the transmission of captured image data. However, in this application example, captured image data is transmitted for tampering determination processing and various analysis processing. Therefore, the captured image data to be transmitted may have a low resolution as long as it is possible to execute tampering determination and analysis processing. In addition, when only tampering determination processing is executed without performing biometric analysis processing or individual analysis processing, captured image data with a low resolution to the extent that a person cannot recognize the subject may be transmitted. This reduces the risk of leakage of captured image data.

[0163] <4. Fourth embodiment> <4-1. System configuration> In the information processing system 1A in the fourth embodiment, tampering determination processing, biometric analysis processing, and individual analysis processing are performed when logging in or making a new registration in order to use various services. FIG. 20 is a block diagram showing an example of a schematic configuration of an information processing system 1A.

[0164] The information processing system 1A includes a user terminal 2, a network 3, and a server device 4. The user terminal 2 is an information processing device used by a user who uses various services (for example, a social networking service).

[0165] As in the first embodiment, the network 3 may be various examples such as the Internet.

[0166] The server device 4 is an information processing device for providing various services, and is provided with, for example, a user information management function, an API (Application Programming Interface) function, and the like. In this embodiment, the server device 4 executes the tampering determination process and various analysis processes.

[0167] An example of the configuration of the user terminal 2 is shown in FIG. The user terminal 2 includes an array sensor 10 for acquiring captured image data that is the subject of an electronic signature, an image processor 11, a control unit 12, a memory unit 13, an input unit 14, a communication unit 15, and an output unit 16. The control unit 12 does not include the units shown in Fig. 2, such as the acquisition unit 31. The array sensor 10, image processor 11, memory unit 13, input unit 14, communication unit 15, and output unit 16 are similar to those described in the first embodiment, and therefore will not be described in detail.

[0168] An example of the configuration of the server device 4 is shown in FIG. The server device 4 includes a control unit 12 , a memory unit 13 , an input unit 14 , a communication unit 15 , and an output unit 16 . The control unit 12 has various functions, specifically, an acquisition unit 31, a decryption processing unit 32, a hash value calculation unit 33, a hash value comparison unit 34, a tampering determination unit 35, a biometric analysis processing unit 36, a biometric determination unit 37, an individual analysis processing unit 38, an individual determination unit 39, an output control unit 40, a status determination unit 41, and an authentication processing unit 42.

[0169] The acquisition unit 31, the decryption processing unit 32, the hash value calculation unit 33, the hash value comparison unit 34, the tampering determination unit 35, the biometric analysis processing unit 36, the biometric determination unit 37, the individual analysis processing unit 38, the individual determination unit 39, the output control unit 40, and the status determination unit 41 are similar to those provided in the user terminal 2 of the first embodiment, and therefore will not be described in detail.

[0170] The authentication processing unit 42 executes authentication processing to determine whether or not the user is allowed to use the service provided by the server device 4. Specifically, the authentication processing unit 42 executes processing to determine whether or not the person using the user terminal 2 matches a registered user. This allows, for example, a determination to be made as to whether or not login is permitted.

[0171] The memory unit 13, input unit 14, communication unit 15 and output unit 16 are similar to those in the first embodiment, and therefore will not be described in detail.

[0172] <4-2.Login screen> When a user performs an operation to display a login screen using the user terminal 2 in order to use a service provided by the server device 4, for example, a login screen 53 shown in FIG. 23 is displayed. In the example shown in FIG. 23, a modal window serving as login screen 53 is superimposed on the originally displayed screen.

[0173] The login screen 53 is provided with, for example, an ID input field 80 for inputting an account name (or a user ID (Identification) or an email address, etc.), a password input field 81 for inputting a password, a guarantee icon 82, a login button 83 for requesting authentication processing from the server device 4, and a cancel button 84 for canceling the login operation.

[0174] The user can cause the server device 4 to execute authentication processing by pressing a login button 83 after inputting a predetermined character string in an ID input field 80 and a password input field 81 .

[0175] The guarantee icon 82 is displayed according to the result of a tamper detection process executed on the server device 4 when an image capturing operation is performed on the user terminal 2 and the captured image data, hash value, and public key are sent to the server device 4 when the login screen 53 is displayed. That is, the guarantee icon 82 is displayed to present to the user what kind of guarantee (such as the degree of guarantee) the server device 4 has performed on the captured image data captured at the user terminal 2.

[0176] The guarantee icon 82 is displayed according to the process contents executed by the server device 4, and when the server device 4 executes only the falsification determination process, either the non-guaranteed icon MK0 or the first guarantee icon MK1 is displayed. Furthermore, when the server device 4 executes not only the tampering determination process but also the liveness determination process or the individual determination process, any one of the non-certified icon MK0, the first certified icon MK1, the second certified icon MK2, and the third certified icon MK3 is displayed.

[0177] <4-3. User terminal processing> An example of the process executed by the control unit 12 of the user terminal 2 in the fourth embodiment will be described with reference to FIG. Note that the same processes as those executed by the user terminal 2 shown in FIG. 9 are denoted by the same reference numerals and descriptions thereof will be omitted as appropriate.

[0178] When an operation for displaying the login screen 53 is performed by the user, the control unit 12 of the user terminal 2 activates the camera function in step S101 and performs an image capturing operation using the array sensor. Next, the control unit 12 of the user terminal 2 generates a hash value from the captured image data in step S102, and encrypts the hash value in step S103.

[0179] In step S121, the control unit 12 of the user terminal 2 transmits the captured image data, the encrypted hash value, and the public key to the server device 4. Note that the captured image data to be transmitted may be encrypted. In that case, the process of step S111 in FIG. 14 may be executed before transmitting the captured image data to the server device 4.

[0180] The control unit 12 of the user terminal 2 determines whether or not a character input operation has been detected in step S122. If a character input operation has been detected, the control unit 12 of the user terminal 2 performs processing to display the input characters in the corresponding input fields (ID input field 80, password input field 81) in step S123. Note that with respect to the password input field 81, a predetermined character (for example, "*") may be displayed in place of the input character string so that it cannot be recognized.

[0181] After executing the process of step S123, the control unit 12 of the user terminal 2 returns to the process of step S122 again.

[0182] If a character input operation has not been detected, the control unit 12 of the user terminal 2 determines whether or not determination result information has been received from the server device 4 in step S124. When a determination process result such as a result of a tampering determination process is received, the control unit 12 of the user terminal 2 performs a process of displaying the guarantee icon 82 on the login screen 53 in step S125. Note that, when a new determination result is received while the guarantee icon 82 is already displayed, the process of step S125 may be a process of updating the guarantee icon 82.

[0183] When the determination result information has not been received from the server device 4, the control unit 12 of the user terminal 2 determines in step S126 whether or not pressing of the login button 83 has been detected. When detecting the pressing of the login button 83, the control unit 12 of the user terminal 2 executes the process of transmitting an authentication process request to the server device 4 in step S127, and then ends the series of processes shown in Fig. 24. When ending the series of processes, the display of the login screen 53 may be ended.

[0184] If pressing of the login button 83 is not detected, the control unit 12 of the user terminal 2 determines in step S128 whether pressing of the cancel button 84 is detected. When detecting that the cancel button 84 has been pressed, the control unit 12 of the user terminal 2 ends the series of processes shown in FIG.

[0185] On the other hand, if pressing of the cancel button 84 is not detected, the control unit 12 of the user terminal 2 returns to the process of step S122 again. That is, the control unit 12 of the user terminal 2 sequentially executes the processes of steps S122, S124, S126, and S128 to check whether it has detected a character input operation, a button press operation, or receipt of a judgment result, and executes the corresponding process (steps S123, S125, S127) if it detects any of the operations.

[0186] <4-4. Processing of server device> An example of the process executed by the server device 4 in the fourth embodiment will be described. Note that the same processes as those shown in FIG. 10 are denoted by the same reference numerals and the description thereof will be omitted as appropriate.

[0187] In response to receiving the captured image data, the encrypted hash value, and the public key from the user terminal 2, the control unit 12 of the server device 4 starts a series of processes shown in FIG. Specifically, the control unit 12 of the server device 4 acquires the captured image data, the encrypted hash value, and the public key in step S201, performs a decryption process of the encrypted hash value in step S202, calculates the hash value from the received captured image data in step S203, performs a comparison process of the hash values ​​in step S204, and performs a tampering determination process in step S205.

[0188] Next, in step S221, the control unit 12 of the server device 4 performs a process of notifying the user terminal 2 of the determination result of the tampering determination process. As a result, a notification as to whether or not it has been guaranteed that the captured image data has not been tampered with is sent to the user terminal 2. In response to this notification, the user terminal 2 displays a guarantee icon 62 on the login screen 53 (processing of step S125 in FIG. 24).

[0189] The control unit 12 of the server device 4 may execute a biometric analysis process and a biometric determination process, an individual analysis process and an individual determination process, etc., in addition to the tampering determination process of step S205. In this case, a notification process is performed according to the results of the tampering determination process, the biometric determination process, and the individual determination process, and the user terminal 2 displays any one of the non-certified icon MK0, the first certified icon MK1, the second certified icon MK2, and the third certified icon MK3 on the login screen 53.

[0190] Returning to the explanation of Figure 25. After performing a process of notifying the determination result in step S221, the control unit 12 of the server device 4 determines whether or not an authentication process request has been received from the user terminal 2 in step S222. If an approval request has not been received, the control unit 12 of the server device 4 determines in step S223 whether a certain time has elapsed since the determination result notification process. If the certain time has elapsed, the control unit 12 of the server device 4 ends the series of processes shown in FIG.

[0191] On the other hand, if the certain period of time has not elapsed, the control unit 12 of the server device 4 returns to the process of step S222.

[0192] If a long time has passed since the result of the tampering determination process was notified, there is a high possibility that the person operating the user terminal 2 has changed. Therefore, in this example, if a predetermined time has passed between the notification of the determination result and the reception of an authentication process request, the series of processes shown in Fig. 25 is terminated so that the authentication process in step S225 described below is not executed. This prevents the subject shown in the captured image data received in step S201 from being different from the person who actually requested the authentication process. In other words, this configuration can reduce the possibility of unauthorized login.

[0193] If it is determined in step S222 that an authentication processing request has been received, the control unit 12 of the server device 4 executes a matching process in step S224. The matching process is, for example, a process of acquiring the account name (user ID) and password contained in the authentication processing request received from the user terminal 2 and matching them with the information stored in the database. If the information received from the user terminal 2 matches the information stored in the database, it is determined that the matching is successful. If they do not match, it is determined that the matching is unsuccessful.

[0194] The control unit 12 of the server device 4 executes authentication processing in step S225. The authentication processing is processing for determining whether or not to allow the user to use a specific function based on the results of the tampering determination processing and the comparison processing. That is, if it is determined that the authentication is successful, the user can use the specific function. On the other hand, if it is determined that the authentication is unsuccessful, the user cannot use the specific function.

[0195] In step S226, the control unit 12 of the server device 4 executes a process of notifying the result of the authentication process. As a result, information indicating whether the authentication (login) was successful or not is displayed on the screen of the user terminal 2. For example, if the authentication is successful, My Page is displayed, and if the authentication is unsuccessful, a page for notifying that the authentication has failed is displayed.

[0196] There are various possible combinations of the results of the tampering determination process and the authentication results. Among them, authentication may be determined to be successful when, for example, the result of the tampering determination is "not tampered" and the result of the authentication process is "successful authentication" (Case 1), or when the result of the tampering determination is "tampered" and the result of the authentication process is "successful authentication" (Case 2).

[0197] In case 1, the various functions provided by the server device 4 can be used without any problems. Regarding Case 2, since the tampering determination result is "tampering determination," various aspects are possible. For example, among the various functions provided by the server device 4, some of the functions may be provided in a restricted state.

[0198] Also, for case 2, it may be configured so that all functions can be used only under certain conditions. Specifically, the MAC address of the user terminal 2 used by the user is acquired, and a process is further executed to determine whether or not it has been used in the past. If it is determined that the user terminal 2 has been used in the past, it is determined that the authentication has been successful. Alternatively, secondary authentication may be performed using the user terminal 2 or another terminal. Specifically, a process may be performed in which an authentication code is sent to a registered email address and the user is prompted to enter the authentication code or to enter an answer to a secret question. If the user is able to perform the correct input operations for these processes, the authentication may be determined to be successful and the user may be permitted to use all functions.

[0199] When the server device 4 executes not only the tampering determination process but also the results of the biometric determination process and the individual determination process, it can decide whether to provide all functions or some of the functions and to execute the secondary authentication process based on a combination of the results of these determination processes. With this configuration, unauthorized or inappropriate use of the functions provided by the server device 4 can be prevented.

[0200] The information processing system 1 according to the fourth embodiment can also be applied to a sign-up system equipped with a tablet terminal or the like that can input a handwritten signature. Specifically, the information processing system 1 is equipped with an operation terminal installed for visitors who come to the company and a server device 4 that performs authentication. 26 shows a specific example of the sign-up screen 54 displayed on the tablet terminal. The sign-up screen 54 displays a signature input field 90 for inputting a signature and a guarantee icon 91 displayed in the signature input field 90. The guarantee icon 91 is displayed according to the content of the process executed by the server device 4, and when the server device 4 executes only the falsification determination process, either the non-guaranteed icon MK0 or the first guarantee icon MK1 is displayed. Furthermore, when the server device 4 executes not only the tampering determination process but also the liveness determination process or the individual determination process, any one of the non-certified icon MK0, the first certified icon MK1, the second certified icon MK2, and the third certified icon MK3 is displayed.

[0201] In that case, the processes executed by the user terminal 2 explained in each figure may be executed by an operation terminal installed for the visitor. That is, the user terminal 2 in the above explanation may be read as the operation terminal. As a result, the operation terminal becomes an information processing device equipped with the array sensor 10 shown in Fig. 21. Also, the server device 4 executes various determination processes related to the captured image data of the visitor captured by the array sensor 10 and the handwritten signature input by the visitor.

[0202] <5. Modifications> In each of the above-described embodiments, the tampering determination process, the liveness determination process, and the individual determination process are executed only in the receiving information processing device. In this modification, the information processing device on the transmission side also executes the above processes. Specifically, the information processing device on the transmission side executes the biometric determination process. An example of the process executed by the user terminal 2 as the information processing device on the transmission side in this case will be described with reference to FIG. 27. In this modification, the information processing system 1 is taken as an example of a video chat system.

[0203] The same processes as those shown in FIG. 9 are denoted by the same reference numerals and the description thereof will be omitted as appropriate. The user terminal 2 performs an image capturing operation in step S101, generates a hash value in step S102, and encrypts the hash value in step S103. In this example, a plurality of captured image data are acquired.

[0204] Next, in step S131, the user terminal 2 performs a process of analyzing the captured image data of a plurality of frames and extracting biometric information. This process is the same as the process of step S211 (FIG. 18) executed by the receiving user terminal 2B in the third embodiment.

[0205] Next, in step S132, the user terminal 2 determines whether or not the subject is a living body based on the extracted biological information. This process is similar to the process in step S212 in FIG.

[0206] In step S133, the user terminal 2 performs branching processing based on the result of the determination as to whether or not the subject is a living body. If it is determined that the subject is a living organism, the user terminal 2 transmits a call start request to the server device 4 in step S104.

[0207] On the other hand, if the subject is not determined to be a living body (or is determined to be a non-living body), the user terminal 2 ends the series of processes shown in Fig. 27. That is, a call start request is not made to the server device 4.

[0208] As a result, the server device 4 receives the call start request only when it is guaranteed that the subject is a living body in the user terminal 2. Therefore, if the server device 4 cannot determine that the subject is a living body, there is a high possibility that a fraudulent call start request has been made by a fraudulent person. In this case, by not starting the call even if a call start request is received, it is possible to prevent fraud and other fraudulent acts in advance.

[0209] Note that the user terminal 2 may be configured to perform individual analysis processing and individual determination processing, and transmit the call start request in step S104 only when information capable of identifying an individual is extracted. This makes it possible to further prevent unauthorized call initiation requests.

[0210] In the above example, the user is notified by the guarantee icon 62, but other methods may be used. For example, the user may be notified by voice. Furthermore, both the notification by the guarantee icon 62 and the notification by voice may be performed simultaneously or in a time-division manner. For example, if the information processing system 1 is a video chat system, it is conceivable that the ringtone can be changed depending on the guarantee state. Also, the appearance or design of the screen may be changed, such as by changing the background color of the incoming call screen 50 depending on the guarantee status.

[0211] At least one of the above-mentioned user terminals 2 may be a store terminal. For example, in order to ensure that a user (customer or an operator in the store) is actually operating the store terminal in front of the terminal, the store terminal may perform an image capturing operation, generate a hash value, and encrypt the hash value, and the server device may perform a decryption process, a hash value calculation process, a hash value comparison process, and a tampering determination process to detect that hacking has not been performed. In this case, the server device may further perform a biometric analysis process and a biometric determination process to ensure that automatic fraudulent acts by a machine have not been performed. Furthermore, the server device may perform an individual analysis process and an individual determination process, and compare the individual feature amount of the acquired operator with data about the operator that has been saved in advance to ensure that the operation is certainly performed by a registered operator. This makes it difficult to perform fraudulent acts, thereby improving security.

[0212] <6. Summary> As described in each of the above-mentioned embodiments and variations, the information processing system 1 includes an array sensor 10 having a pixel array 21 in which multiple pixels having visible light or invisible light receiving elements are arranged in one or two dimensions, a hash value generation unit 22 that generates a hash value from captured image data obtained based on photoelectric conversion in the pixel array, and an encryption processing unit 23 that performs encryption processing on the hash value, an acquisition unit 31 that acquires the captured image data and the encrypted hash value, a decryption processing unit 32 that decrypts the acquired encrypted hash value, a hash value calculation unit 33 that calculates a hash value from the acquired captured image data, a hash value comparison unit 34 that compares the decrypted hash value with the calculated hash value, and a tampering determination unit 35 that determines whether the acquired captured image data has been tampered with based on the comparison result of the hash values. That is, the generation of the hash value and the encryption process are performed within the array sensor 10 . This significantly reduces the possibility that captured image data will be tampered with during the period from when the captured image data is captured until when the hash value is generated, making it more difficult to tamper with the captured image data and improving the performance of detecting tampering.

[0213] As described in the first and third embodiments, the device may be equipped with a biometric analysis processing unit 36 ​​that performs analysis to extract biometric information of the subject based on the acquired captured image data, and a biometric determination unit 37 that determines whether or not the subject is a living organism based on the analysis results of the biometric analysis processing unit 36. This makes it possible to determine whether the subject is a living organism or not. Therefore, for example, it is possible to determine whether a call start request or authentication processing request has been sent when a living human is in front of the terminal, and it is possible to distinguish between a legitimate login operation by a user and an unauthorized login operation by a machine, thereby preventing fraudulent acts, etc. In other words, for example, by placing a 3D printed model of a face in front of the camera, it will be difficult to mistakenly believe that the procedure is being carried out by the actual person. Also, for example, on a website or the like that has a chat function for inquiries, when the chat function is used and an operator is actually responding, a guarantee mark that guarantees that a human is responding can be displayed on the user terminal used by the inquiring user, thereby emphasizing that the response is not mechanical and increasing the user's credibility. In this case, the user terminal used by the operator is the user terminal 2A in the third embodiment that performs the hash value generation process and the hash value encryption process, and the user terminal used by the inquiring user is the user terminal 2B in the third embodiment that performs the decryption process, the hash value calculation process, the hash value comparison process, the tampering determination process, the biometric analysis process, and the biometric determination process.

[0214] As described in the first and third embodiments, the device may be provided with an individual analysis processing unit 38 for analyzing individual features of the subject contained in the acquired captured image data, and an individual determination unit 39 for determining whether or not the subject can be recognized as an individual based on the analysis results of the individual analysis processing unit 38. This makes it possible to obtain information for identifying the individual subject. Therefore, for example, it becomes possible to provide functions to specific users and to notify the other party of a communication that the user is a specific user, thereby preventing tampering, impersonation, etc., and preventing fraud, etc. In addition, for example, a user engaged in sales activities can use an imaging device having an array sensor 10 capable of executing the hash value generation process and encryption process of the present technology to make a call request to another user, thereby reducing the sense of caution of the other users and increasing their trust in the user, making it easier to conduct sales activities.

[0215] As explained in the first embodiment and the like, an output control unit 40 may be provided that performs output control for notifying the determination result of the falsification determination unit 35. This allows the user to recognize the tampering determination result. Therefore, the user can select an operation according to the judgment result. For example, in the case of a video chat system, the user can select whether or not to answer an incoming call from the other party based on the tampering judgment result.

[0216] As explained in the third embodiment and the like, an output control unit 40 that performs output control for notifying the determination result of the liveness determination unit 37 may be provided. This makes it possible to notify the user of the liveness determination result. Therefore, the user can select an operation according to the result of the biometric determination. For example, it becomes possible to distinguish an automatic call request by a machine and to reject such a call request.

[0217] As explained in the third embodiment, an output control unit 40 may be provided that performs output control for notifying the determination result of the individual determination unit 39. This makes it possible to notify the user of the results of the individual determination. Therefore, the user can select an operation according to the result of the individual determination, for example, it becomes possible to reject a call request from a suspicious user who has not been personally authenticated.

[0218] As described in each embodiment, the system is configured to include an imaging device (user terminal 2A) and an information processing device (user terminal 2B), where the imaging device has an array sensor 10, and the information processing device may have an acquisition unit 31, a decoding processing unit 32, a hash value calculation unit 33, a hash value comparison unit 34, a tampering determination unit 35, and an output control unit 40. For example, the present invention can be applied to a surveillance camera system that is made up of an imaging device and an information processing device. Specifically, the server device serving as the information processing device can confirm that the image data captured by the surveillance camera is indeed data captured by the surveillance camera and has not been tampered with. Furthermore, in the case of a two-way call, the user terminal 2 (2A) used by one user can be regarded as an imaging device, and the user terminal 2 (2B) used by the other user can be regarded as an information processing device.

[0219] As described in each embodiment, the output control unit 40 may execute control to display and output the acquired captured image data, and control to display and output the judgment result of the tamper judgment unit 35 regarding the acquired captured image data. One example of such an information processing system is one that performs user interface control to display a received captured image while indicating whether or not the image has been tampered with. For example, a video chat system or a security camera management system may be considered. In an auction system, when an image of an auctioned product is displayed in an application (auction app) used by a user, the captured image data (data of the product image) uploaded to a server device and an encrypted hash value may be acquired, a process of determining whether the image is tamper-proof may be performed, and the determination result may be displayed together with the image. This allows the user to confirm that the image is not a tampered image in which scratches on the product have been erased using an image processing application or the like, improving convenience. It may also be possible to distinguish fraudulent acts such as unfairly making a product appear to be of high quality. Specifically, on an auction site in an auction system or a search site in an accommodation search system, images of items for sale (see FIG. 28) and images of rooms (see FIG. 29) are posted. An icon indicating that the images have not been tampered with is superimposed on the top of the captured images. By superimposing such an icon indicating that the images have not been tampered with using an image processing application or the like, the user can confirm that the images are not tampered with images in which the subject (item for sale or room) has been improved in appearance. Such a subject is not a subject from which biometric information can be extracted as described above, but an inorganic subject from which biometric information cannot be extracted, etc. In other words, the present technology can be applied not only to subjects such as people from which biometric information can be extracted, but also to subjects such as inorganic objects.

[0220] As described in the application example of the third embodiment, the output control unit 40 may perform control to display and output the determination result of the tamper determination unit 35 without displaying the acquired captured image data. An example of such an information processing system is a text chat system in which captured images are not displayed. Even if an image is not displayed, it is possible to provide an index for determining whether the communication partner is trustworthy or not, thereby making it possible to prevent fraud and the like.

[0221] As described in the application example of the third embodiment, the output control unit 40 may perform control to present a chat screen and control to display and output, for each chat participant, on the chat screen, the judgment results of the tampering judgment unit 35 regarding the captured image data of the chat participants received from the imaging device (user terminal 2A) used by the chat participants. In a text chat system where captured images are not displayed, it is difficult to know what kind of people are using the chat. With this configuration, an indicator of whether each participant is trustworthy is displayed. Therefore, the users can communicate appropriately.

[0222] As described in the fourth embodiment, the system is configured to include an imaging device (user terminal 2) and an information processing device (server device 4), where the imaging device has an array sensor 10 and an output control unit 40, and the information processing device may have an authentication processing unit 42 that performs authentication processing, an acquisition unit 31, a decryption processing unit 32, a hash value calculation unit 33, a hash value comparison unit 34, and a tampering determination unit 35. As an example of such an information processing system, a system in which an authentication result or the like is notified in an information processing device (imaging device) that has transmitted captured image data is considered. Specifically, there is an authentication system that includes an authentication target terminal (such as a smartphone) that is an imaging device, and a server device that performs authentication processing. This allows the server device to distinguish fraudulent authentication requests, and prevents authentication from being performed in response to a fraudulent authentication request.

[0223] As described in the fourth embodiment, the output control unit 40 may control the display output of the judgment result of the tampering judgment unit 35 obtained from the information processing device (server device 4), and the authentication processing unit 42 may perform authentication processing based on the judgment result of the tampering judgment unit 35. That is, in the device that transmitted the captured image data (imaging device: user terminal 2), the user can recognize whether the transmitted captured image data reached the receiving device, the server device 4 (information processing device), without being tampered with. This allows the user to recognize whether or not they are currently being hacked. Moreover, it becomes easier for the server device 4 to distinguish unauthorized authentication requests (login requests, etc.), making it possible to achieve stricter authentication.

[0224] As described in the fourth embodiment, the information processing device (server device 4) includes a biometric analysis processing unit 36 ​​that performs analysis to extract biometric information of the subject based on the acquired captured image data, and a biometric determination unit 37 that determines whether the subject is a living body or not based on the analysis result of the biometric analysis processing unit 36, an output control unit 40 controls the display output of the determination result of the biometric determination unit 37, and an authentication processing unit 42 may perform authentication processing based on the determination result of the biometric determination unit 37. That is, in addition to the tampering determination process, the image processing device extracts biometric information from the received captured image data and performs a process of determining whether or not the subject is a living body based on the extraction result. This makes it possible to make authentication more strict and improve security.

[0225] As described in the first and third embodiments, the image capturing apparatus may include an individual analysis processing unit 38 for analyzing individual features of the subject contained in the acquired captured image data, an individual determination unit 39 for determining whether or not the subject can be recognized as an individual based on the analysis results of the individual analysis processing unit 38, and a state determination unit 41 for determining, based on the determination results as to whether or not the captured image data has been tampered with, the determination results as to whether or not the subject is a living organism, and the determination results as to whether or not the individual can be recognized, a non-secure state ST0 in which it is not guaranteed that the acquired captured image data has not been tampered with, a first secured state ST1 in which it is guaranteed that the acquired captured image data has not been tampered with, a second secured state ST2 in which it is guaranteed that the acquired captured image data has not been tampered with and the subject is determined to be a living organism, or a third secured state ST3 in which it is guaranteed that the acquired captured image data has not been tampered with, the subject is determined to be a living organism, and individual recognition is possible. This makes it possible to carry out processing according to the degree of security assurance. For example, an image may be output according to the degree of security, or functions may be restricted according to the degree of security. For example, in a server device that performs authentication processing, it is possible to perform processing that permits the use of all functions in the third security state ST3 and to permit the use of some functions with restrictions in the second security state ST2.

[0226] As described in the first and third embodiments, an output control unit 40 may be provided that performs display output according to the non-secured state ST0, the first secured state ST1, the second secured state ST2, and the third secured state ST3. This allows the user to check the degree of guarantee. For example, in a system in which communication between users takes place, such as a chat system or a telephone call system, the user can select an operation according to the degree of assurance. Furthermore, when a display output is performed in the server device 4, the administrator or the like can take measures according to the display output.

[0227] As described in the third embodiment, the living body determination section 37 may determine whether or not the subject is a living body by determining the presence or absence of a change in the biological signal of the subject obtained as a result of the analysis. For example, a change in a biological signal is extracted by analyzing a plurality of captured image data. Examples of biosignals include the presence or absence of nystagmus, lip sync, pupil reaction, heart rate detection by spectrum analysis, etc. By performing such quantitative analysis, it is possible to determine, for example, whether or not a human operation has been performed, and to prevent fraudulent acts and unauthorized access by machines.

[0228] The program of the embodiment is a program that causes, for example, a CPU, a DSP, or a device including these to execute the processes shown in each of Figures 9, 10, 14 to 18, and 24 to 27. That is, the program of the embodiment causes the control unit 12 to execute a process of acquiring captured image data and an encrypted hash value from an array sensor having a pixel array in which multiple pixels having visible light or invisible light receiving elements are arranged one-dimensionally or two-dimensionally, a hash value generation unit that generates a hash value from captured image data obtained by photoelectric conversion, and an encryption processing unit that performs encryption processing on the hash value. The control unit 12 also executes a process for decrypting the obtained encrypted hash value. Furthermore, the control unit 12 is caused to execute a process of calculating a hash value from the acquired captured image data. Furthermore, the control unit 12 is caused to execute a process of comparing the decrypted hash value with the calculated hash value. In addition, the control unit 12 is caused to execute a process of determining whether or not the acquired captured image data has been tampered with based on the comparison result of the hash values. The above-mentioned information processing system 1 can be realized by such a program.

[0229] A program for realizing such an information processing system 1 can be recorded in advance in a HDD serving as a recording medium built into a device such as a computer device, or in a ROM or the like in a microcomputer having a CPU. Alternatively, the software may be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, a memory card, etc. Such removable recording media may be provided as a so-called package software. Furthermore, such a program can be installed in a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0230] Furthermore, such a program is suitable for widespread provision of the information processing system 1 (or the user terminal 2 therein) of the embodiment. For example, by downloading the program to a mobile terminal device such as a smartphone or tablet, a mobile phone, a personal computer, a game device, a video device, a PDA (Personal Digital Assistant), or other device equipped with a camera function, the smartphone or the like can function as the information processing system 1 (or the user terminal 2) of the present disclosure.

[0231] Furthermore, such a program can be used to control the user interface in devices equipped with a camera function, such as mobile terminal devices such as smartphones and tablets, mobile phones, personal computers, game devices, video devices, and PDAs.

[0232] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0233] <7. This Technology> (1) an array sensor including a pixel array in which a plurality of pixels, each having a light receiving element for receiving visible light or invisible light, are arranged one-dimensionally or two-dimensionally, a hash value generating unit that generates a hash value from captured image data obtained by photoelectric conversion, and an encryption processing unit that performs encryption processing on the hash value; an acquisition unit that acquires the captured image data and the encrypted hash value; a decryption processing unit that decrypts the acquired encrypted hash value; a hash value calculation unit that calculates a hash value from the acquired captured image data; a hash value comparison unit that compares the decrypted hash value with the calculated hash value; and a tampering determination unit that determines whether or not the acquired captured image data has been tampered with based on a comparison result of the hash values. Information processing system. (2) a biometric analysis processing unit that performs an analysis to extract biometric information of a subject based on the acquired captured image data; a biological analysis unit that determines whether the subject is a biological subject based on the analysis result of the biological analysis processing unit. The information processing system according to (1) above. (3) an individual analysis processing unit for analyzing individual feature amounts of a subject included in the acquired captured image data; an individual determination unit that determines whether or not the subject can be recognized as an individual based on the analysis result of the individual analysis processing unit; An information processing system according to any one of (1) to (2) above. (4) An output control unit that performs output control for notifying the determination result of the tampering determination unit. An information processing system according to any one of (1) to (3) above. (5) an output control unit that performs output control for notifying a result of the determination by the liveness determination unit; The information processing system according to (2) above. (6) an output control unit that performs output control for notifying a determination result of the individual determination unit; The information processing system according to (3) above. (7) The imaging device and the information processing device are included. The imaging device includes the array sensor, The information processing device includes the acquisition unit, the decryption processing unit, the hash value calculation unit, the hash value comparison unit, the tampering determination unit, and the output control unit. The information processing system according to (4) above. (8) The output control unit executes control to display and output the acquired captured image data, and control to display and output a determination result of the tamper determination unit regarding the acquired captured image data. The information processing system according to (7) above. (9) The output control unit controls displaying and outputting a determination result of the tamper determination unit with respect to the acquired captured image data. The information processing system according to (7) above. (10) The output control unit is A control for presenting a chat screen; and controlling the display output of the determination result of the tamper determination unit for the captured image data of the chat participants received from the imaging device used by the chat participants on the chat screen for each chat participant. The information processing system according to (9) above. (11) The imaging device and the information processing device are included. The imaging device includes the array sensor and the output control unit, The information processing device includes an authentication processing unit that performs authentication processing, the acquisition unit, the decryption processing unit, the hash value calculation unit, the hash value comparison unit, and the tampering determination unit. The information processing system according to (4) above. (12) the output control unit controls displaying and outputting a determination result of the tampering determination unit acquired from the information processing device; The authentication processing unit performs the authentication process based on a determination result of the tampering determination unit. The information processing system according to (11) above. (13) The information processing device includes: a biometric analysis processing unit that performs an analysis to extract biometric information of a subject based on the acquired captured image data; a living body determination unit that determines whether the subject is a living body based on an analysis result of the living body analysis processing unit, The output control unit controls to display and output a determination result of the liveness determination unit, The authentication processing unit performs the authentication process based on a determination result of the biometric determination unit. The information processing system according to (12) above. (14) an individual analysis processing unit for analyzing individual feature amounts of a subject included in the acquired captured image data; an individual determination unit that determines whether or not the subject can be recognized as an individual based on the analysis result of the individual analysis processing unit; and a state determination unit that determines, based on the determination result on the presence or absence of tampering, the determination result on whether the subject is a living organism, and the determination result on whether individual recognition is possible, a non-guaranteed state in which it is not guaranteed that the acquired captured image data has not been tampered with, a first guaranteed state in which it is guaranteed that the acquired captured image data has not been tampered with, a second guaranteed state in which it is guaranteed that the acquired captured image data has not been tampered with and the subject is determined to be a living organism, and a third guaranteed state in which it is guaranteed that the acquired captured image data has not been tampered with, the subject is determined to be a living organism, and the individual recognition is possible. The information processing system according to (2) above. (15) an output control unit that performs display output according to the non-secured state, the first secured state, the second secured state, and the third secured state; The information processing system according to (14) above. (16) The living body determination unit determines whether or not the living body is present by determining whether or not there is a change in the biological signal obtained as a result of the analysis. The information processing system according to (2) above. (17) The subject in the captured image data is a subject from which biological information cannot be extracted. The information processing system according to (1) above. (18) a process of acquiring the captured image data and the encrypted hash value from an array sensor having a pixel array in which a plurality of pixels, each having a light receiving element for visible light or non-visible light, are arranged one-dimensionally or two-dimensionally, a hash value generating unit that generates a hash value from captured image data obtained by photoelectric conversion, and an encryption processing unit that performs encryption processing on the hash value; A process of decrypting the obtained encrypted hash value; A process of calculating a hash value from the acquired captured image data; comparing the decrypted hash value with the calculated hash value; A process of determining whether or not the acquired captured image data has been tampered with based on a comparison result of the hash values. An information processing method executed by an information processing device. (19) a process of acquiring the captured image data and the encrypted hash value from an array sensor having a pixel array in which a plurality of pixels, each having a light receiving element for visible light or non-visible light, are arranged one-dimensionally or two-dimensionally, a hash value generating unit that generates a hash value from captured image data obtained by photoelectric conversion, and an encryption processing unit that performs encryption processing on the hash value; A process of decrypting the obtained encrypted hash value; A process of calculating a hash value from the acquired captured image data; comparing the decrypted hash value with the calculated hash value; A process of determining whether or not the acquired captured image data has been tampered with based on a comparison result of the hash values. A program to be executed by an information processing device. (20) The array sensor includes a pixel array in which a plurality of pixels having light receiving elements for visible or invisible light are arranged one-dimensionally or two-dimensionally, a hash value generating unit that generates a hash value from captured image data obtained by photoelectric conversion, and an encryption processing unit that performs encryption processing on the hash value. ... notifies the result of a determination process as to whether or not the captured image data has been tampered with, the determination process being performed based on a comparison result between a hash value obtained by decrypting the encrypted hash value obtained from the array sensor and a hash value calculated from the captured image data obtained from the array sensor. User interface. [Explanation of symbols]

[0234] 1,1A Information Processing System 2, 2A, 2B User terminal 4. Server equipment 10 Array Sensor 12 Control section 21 Pixel Array 22 Hash value generator 23 Encryption processing section 24 Image encryption processing section 31 Acquisition Department 32 Decoding processing section 33 Hash value calculation unit 34 Hash value comparison section 35 Tampering Detection Unit 36 Bioanalysis Processing Unit 37 Biometrics Unit 38 Individual Analysis Processing Unit 39 Individual Identification Section 40 Output control section 41 Status determination unit 42 Authentication processing section

Claims

1. an array sensor including a pixel array in which a plurality of pixels, each having a light receiving element for receiving visible light or invisible light, are arranged one-dimensionally or two-dimensionally, a hash value generating unit that generates a hash value from captured image data obtained based on photoelectric conversion in the pixel array, and an encryption processing unit that performs encryption processing on the hash value; an acquisition unit that acquires the captured image data and the encrypted hash value; a decryption processing unit that decrypts the acquired encrypted hash value; a hash value calculation unit that calculates a hash value from the acquired captured image data; a hash value comparison unit that compares the decrypted hash value with the calculated hash value; a tampering determination unit that determines whether or not the acquired captured image data has been tampered with based on a comparison result of the hash values; an output control unit that performs output control for notifying a determination result of the tampering determination unit, The output control unit performs control to present a chat screen, On the chat screen, the captured image data of the chat participants output from another imaging device used by the chat participants and having the array sensor is not displayed, and the determination result of the tamper determination unit relating to the captured image data is displayed for each chat participant. Information processing system.

2. a biometric analysis processing unit that performs an analysis to extract biometric information of a subject based on the acquired captured image data; a biological analysis unit that determines whether the subject is a biological subject based on the analysis result of the biological analysis processing unit. The information processing system according to claim 1 .

3. an individual analysis processing unit for analyzing individual feature amounts of a subject included in the acquired captured image data; an individual determination unit that determines whether or not the subject can be recognized as an individual based on the analysis result of the individual analysis processing unit; The information processing system according to claim 1 .

4. The output control unit performs output control for notifying a determination result of the liveness determination unit. The information processing system according to claim 2 .

5. The output control unit performs output control for notifying a determination result of the individual determination unit. The information processing system according to claim 3 .

6. The imaging device and the information processing device are included. The imaging device includes the array sensor, The information processing device includes the acquisition unit, the decryption processing unit, the hash value calculation unit, the hash value comparison unit, the tampering determination unit, and the output control unit. The information processing system according to claim 1 .

7. The output control unit executes control to display and output the acquired captured image data, and control to display and output a determination result of the tamper determination unit regarding the acquired captured image data.

7. The information processing system according to claim 6.

8. The imaging device and the information processing device are included. The imaging device includes the array sensor and the output control unit, The information processing device includes an authentication processing unit that performs authentication processing, the acquisition unit, the decryption processing unit, the hash value calculation unit, the hash value comparison unit, and the tampering determination unit. The information processing system according to claim 1 .

9. the output control unit controls displaying and outputting a determination result of the tampering determination unit acquired from the information processing device; The authentication processing unit performs the authentication process based on a determination result of the tampering determination unit. The information processing system according to claim 8.

10. The information processing device includes: a biometric analysis processing unit that performs an analysis to extract biometric information of a subject based on the acquired captured image data; a living body determination unit that determines whether the subject is a living body based on an analysis result of the living body analysis processing unit, The output control unit controls to display and output a determination result of the liveness determination unit, The authentication processing unit performs the authentication process based on a determination result of the biometric determination unit. The information processing system according to claim 9.

11. an individual analysis processing unit for analyzing individual feature amounts of a subject included in the acquired captured image data; an individual determination unit that determines whether or not the subject can be recognized as an individual based on the analysis result of the individual analysis processing unit; and a state determination unit which determines, based on the determination result on the presence or absence of tampering, the determination result on whether the subject is a living organism, and the determination result on whether individual recognition is possible, a non-guaranteed state in which it is not guaranteed that the acquired captured image data has not been tampered with, a first guaranteed state in which it is guaranteed that the acquired captured image data has not been tampered with, a second guaranteed state in which it is guaranteed that the acquired captured image data has not been tampered with and the subject is determined to be a living organism, and a third guaranteed state in which it is guaranteed that the acquired captured image data has not been tampered with, the subject is determined to be a living organism, and the individual recognition is possible. The information processing system according to claim 2 .

12. The output control unit performs display output according to the non-secured state, the first secured state, the second secured state, and the third secured state. The information processing system according to claim 11.

13. The living body determination unit determines whether or not the subject is a living body by determining whether or not there is a change in a biological signal of the subject obtained as a result of the analysis. The information processing system according to claim 2 .

14. The subject in the captured image data is a subject from which biological information cannot be extracted. The information processing system according to claim 1 .

15. a process of acquiring the captured image data and the encrypted hash value from an array sensor having a pixel array in which a plurality of pixels, each having a light receiving element for visible light or non-visible light, are arranged one-dimensionally or two-dimensionally, a hash value generating unit that generates a hash value from captured image data obtained by photoelectric conversion, and an encryption processing unit that performs encryption processing on the hash value; A process of decrypting the obtained encrypted hash value; A process of calculating a hash value from the acquired captured image data; comparing the decrypted hash value with the calculated hash value; A process of determining whether or not the acquired captured image data has been tampered with based on a comparison result of the hash values; A process of notifying the result of the determination of the presence or absence of tampering; a process of presenting a chat screen in which the captured image data of the chat participants output from another imaging device used by the chat participants and having the array sensor is not displayed, and the determination result of the presence or absence of tampering related to the captured image data is displayed for each chat participant; An information processing method executed by an information processing device.

16. a process of acquiring the captured image data and the encrypted hash value from an array sensor having a pixel array in which a plurality of pixels, each having a light receiving element for visible light or non-visible light, are arranged one-dimensionally or two-dimensionally, a hash value generating unit that generates a hash value from captured image data obtained by photoelectric conversion, and an encryption processing unit that performs encryption processing on the hash value; A process of decrypting the obtained encrypted hash value; A process of calculating a hash value from the acquired captured image data; comparing the decrypted hash value with the calculated hash value; A process of determining whether or not the acquired captured image data has been tampered with based on a comparison result of the hash values; A process of notifying the result of the determination of the presence or absence of tampering; a process of presenting a chat screen in which the captured image data of the chat participants output from another imaging device used by the chat participants and having the array sensor is not displayed, and the determination result of the presence or absence of tampering related to the captured image data is displayed for each chat participant; A program to be executed by an information processing device.

Images