Information processing systems and programs

Abstract

To provide an information processing system which can achieve reduction of time and effort of a user, personal verification performed only for a required action, and prevention of a fraud due to switching of an operator after log-in.SOLUTION: An information processing system, such as an electronic seal system 10, is provided with: image display means 21 which displays, on a screen, an object to be used for drag-and-drop operation of a user in order to carry out a purposeful act; drag operation determination means 61 which determines whether or not the object is being dragged; authentication information acquisition means 62 which acquires, during drag operation, authentication information to be used for determining whether or not to permit drop operation to complete the purposeful act; and authentication means 32 which determines whether or not the user who has performed drag operation is authenticated, using the acquired authentication information and registered user information registered in advance on user information storage means 41.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing system and a program configured by a computer that executes a process of accepting a target action intended by a user through an operation on a screen. For example, it can be used in an electronic seal system, a problem system for language learning and language proficiency tests, or an asset transfer system that executes processes such as transfers to accounts of financial institutions, transfers between accounts, trading of securities or other financial products, e-commerce transactions, actions involving charging in games, or other processes involving asset transfers.

Background Art

[0002] Generally, regardless of the scale of the system, when using a system, login is performed using a login ID. At this time, usually, an authentication process for verifying the identity using a password or the like is executed. In recent years, biometric authentication using biometric information such as faces and fingerprints has also been actively carried out. In addition to the authentication process using a login ID and password, multi-factor authentication that allows users to input information that only they know is also widely used.

[0003] Furthermore, in the case of logging into financial institutions, in addition to authentication processing by entering a login ID and password, a human-to-machine detection process is also performed as part of the user authentication process during login to confirm that the input is human, i.e., to eliminate machine input. This process involves having the user drag multiple jigsaw puzzle pieces on the screen to a designated location (the correct location) and drop them. Also, when accepting operations from an unspecified number of people, such as downloading data from a site, and it is desired to eliminate machine operation, although this is not user authentication during login, processes are used to distinguish between humans and machines, such as having the user select an image containing a specific object (e.g., a traffic light, a crosswalk, etc.) from multiple images displayed on the screen using a checkbox, or having the user type in numbers with a very distorted shape drawn on the screen. Note that the present invention utilizes drag-and-drop operations, but the method of use differs from that of the jigsaw puzzle described above. In addition, the present invention can utilize a user operation to select a specified image from multiple images displayed on the screen for user authentication, but this is different from the image selection operation used for human-to-machine detection described above.

[0004] Furthermore, the present invention can be applied to an electronic seal system. As an electronic seal, for example, there is a known type in which an electronic seal equipped with an RF tag (not a display on a screen, but a physical object) is brought close to a terminal equipped with a touch panel display and an RF tag reader, and the terminal detects the RF tag with the RF tag reader (see Patent Document 1). Also, there is an electronic approval device that, when the approver is authenticated as the person in question, opens a protective file that protects the impression data of the approver's electronic seal, and if it opens successfully, pastes the impression data of the approver's electronic seal protected in the protective file (see Patent Document 2).

[0005] Furthermore, while facial recognition can be applied as an authentication means in this invention, known systems for performing facial recognition include information processing devices that perform facial recognition when making electronic contracts (see Patent Document 3) and systems that use facial recognition in the approval process for elevator maintenance and inspection reports (see Patent Document 4). It should be noted that while Patent Documents 3 and 4 include descriptions of drag-and-drop operations, these differ from the method of using drag-and-drop operations in this invention.

[0006] Furthermore, while the present invention enables identity authentication using hierarchical spatial images, the concept of hierarchizing spatial images itself is already known and has been used in dictionary systems (see Patent Document 5). [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2019-215596 [Patent Document 2] Japanese Patent Publication No. 2020-135690 [Patent Document 3] Japanese Patent Publication No. 2021-117995 (paragraph

[0082] ) [Patent Document 4] Japanese Patent Publication No. 2018-43836 (paragraph

[0054] ) [Patent Document 5] Patent Publication No. 4094283 [Overview of the project] [Problems that the invention aims to solve]

[0008] As mentioned earlier, when logging into the system, authentication is performed using a login ID and password, or multi-factor authentication that adds another layer of authentication. However, all of the operations required of the user for these authentications are operations performed for the purpose of logging in. Similarly, the drag-and-drop operation used to solve the jigsaw puzzle mentioned earlier is not a game, but rather an operation performed as part of the authentication process, and therefore is an operation performed for the purpose of logging in. In other words, these login operations are separate from the operations performed by the user to carry out the intended activity after logging in. Consequently, this increases the effort required of the user.

[0009] Furthermore, the actions users perform within the system after logging in include both important and simple actions that do not require much attention. Therefore, since the aforementioned user authentication during login takes place at the system's entry point, users often log in with the same level of effort as they would for important actions, even when performing simple actions. In other words, regardless of what actions a user performs after logging into the system, the same level of burden is often placed on the user at the system's entry point.

[0010] Furthermore, even if user authentication is performed when logging into the system, if the user leaves their desk for any reason and goes away from the computer, it is impossible to prevent someone else from operating the computer and committing fraudulent activities because the user's authentication has already been completed.

[0011] The objective of the present invention is to provide an information processing system and program that can reduce the burden on users, perform user authentication only for necessary actions, and prevent fraud due to changes in the operator after login. [Means for solving the problem]

[0012] The present invention relates to an information processing system comprising a computer that performs processing to accept the user's intended action through on-screen operations, A user information storage means that stores registered personal information for identity verification, pre-registered by the user, in association with user identification information, A screen display means that performs the process of displaying on the screen objects used for drag-and-drop operations to perform a user's intended action, A means for determining whether or not an object is being dragged performs a process to determine whether or not the object is being dragged, The drag operation determination means determines that a drag operation is in progress, and while this determination is maintained, the authentication information acquisition means executes a process to acquire authentication information used to determine whether or not to permit a drop operation that completes the intended action without requiring any operation other than the drag operation by the user. This authentication means uses the authentication information obtained by this authentication information acquisition means and the registered personal information stored in the user information storage means in association with the user identification information used by the user who performed the drag operation when logging in to perform an authentication process to determine whether the user who performed the drag operation is the person corresponding to the user identification information used when logging in. It is characterized by having the following features.

[0013] Here, the phrase "without requiring any operation other than dragging by the user" in the "means for obtaining authentication information" means that the user does not need to perform any operations other than dragging, as this is solely for the purpose of obtaining authentication information. Therefore, it does not interfere with other operations performed by the user during dragging, such as changing the computer's output volume or screen brightness during dragging. This is because such computer volume operations are performed independently of obtaining authentication information. Furthermore, adjusting the microphone volume to an appropriate volume for the user's (examinee or meeting participant, etc.) voice input during online exams or online meetings is not a volume operation for obtaining authentication information.

[0014] In the information processing system of the present invention, when a user performs a drag operation to carry out a target action, authentication information is acquired during that drag operation. This authentication information is then used to determine whether or not to permit a drop operation to complete the target action.

[0015] In this process, the user is not required to perform any operations other than dragging to obtain authentication information. Furthermore, since the user is performing the dragging operation in order to perform the intended action, they are not being forced to perform the dragging operation solely for the purpose of obtaining authentication information. In other words, the user logs into the system in order to complete the intended action, and the dragging operation is an essential operation for performing that action. Therefore, the authentication information is obtained while the user is performing such an essential operation. This makes it possible to reduce the effort required of the user.

[0016] Furthermore, authentication information is acquired during the drag operation to perform the intended action, which is used to determine whether or not to allow the drop operation to complete the intended action. Therefore, if no drag operation is performed, authentication information is not acquired, and authentication information is only acquired when a drag operation is performed. Thus, unlike the user authentication process performed when logging into the system, which is performed uniformly regardless of what actions the user takes, it is possible to perform the user authentication process only for necessary actions. Consequently, it is possible to avoid performing the same user authentication process for simple, less important actions as for important actions, thus reducing the burden on the user.

[0017] Furthermore, since authentication information is obtained during the drag operation to perform the intended action and user authentication is performed, it becomes possible to prevent fraudulent activity due to changes in the operator after login, thereby achieving the aforementioned objectives.

[0018] Note that the authentication using the authentication information obtained during the drag operation of the present invention may be a part of multi-factor authentication performed in addition to the authentication using a user ID, password, etc. at the time of login, or may be the authentication performed when authentication is not performed at the time of login.

[0019] Also, the jigsaw puzzle by the above-described drag-and-drop operation is performed as part of authentication and has no other purpose. However, the drag-and-drop operation in the present invention is an operation performed only for the purpose of accomplishing the target action intended by the user, and the acquisition of authentication information is only incidentally performed during that operation. Therefore, the usage methods of the two drag-and-drop operations are different. Also, in the case of the jigsaw puzzle, discriminative information between humans and machines indicating that it was possible to drop at a specific location (the correct location) can be obtained only after performing the drop operation. In contrast, in the present invention, the acquisition of authentication information is performed during the drag operation, not after the drop operation.

[0020] Furthermore, although Patent Documents 3 and 4 described above include descriptions of drag-and-drop operations, the acquisition of authentication information is not performed during the drag operation. Therefore, it is different from the usage method of the drag-and-drop operation in the present invention.

[0021] <Specific aspects of the target action> The target actions accomplished by the drag-and-drop operation of the present invention include various actions, and typical target actions are as follows. (1) The act of affixing an electronic seal (2) The act of answering questions such as word rearrangement problems and fill-in-the-blank problems (3) The execution act that causes asset transfers such as transfers, transfers between accounts, and buying and selling transactions

[0022] <Specific aspects of the authentication information obtained during the drag operation> The authentication information obtained during the drag operation of the present invention includes various information, and typical authentication information is as follows. (A) Facial image data used for facial recognition (B) Voice data used for voiceprint authentication (C) The pass-through determination result of whether the drag operation instruction position of the dragged object (mouse or touchpad cursor position, touch panel contact position of finger, etc.) has passed through the image data or drawing target area specified by the user.

[0023] <Arbitrary nature of the combination of intended action and authentication information> For any of the above-mentioned actions for which the purpose is to (1) affix a seal, (2) answer a question, or (3) execute an action that results in the transfer of assets, any of the above-mentioned authentication methods—(A) facial recognition, (B) voiceprint recognition, or (C) authentication based on the pass / fail judgment—can be applied. Furthermore, for any of the above-mentioned actions for which the purpose is to (1), (2), or (3), any combination of authentication methods (A), (B), or (C) can be applied.

[0024] <(A) In the case of facial recognition> In the aforementioned information processing system, The user information storage means is, The system includes a facial recognition information storage means that stores the registered personal information, such as the personal's facial image data or facial features extracted from this facial image data, in association with user identification information. The means of obtaining authentication information is, The system includes a face image acquisition means that performs a process to acquire face image data of the user during a drag operation using a camera as authentication information. The authentication method is, A facial feature extraction means that performs a process to extract facial feature quantities of a user during a drag operation from facial image data acquired by a facial image acquisition means, or in addition to this process, performs a process to extract facial feature quantities of a person from the facial image data of a person if the person's facial image data is stored as registered personal information in a facial recognition information storage means. A face feature comparison means performs a process to compare the face features of the user during the drag operation extracted by this face feature extraction means with the face features of the user stored as registered personal information in the face recognition information storage means, or the face features of the user extracted by the face feature extraction means from the user's face image data stored as registered personal information in the face recognition information storage means, and calculates a face feature score indicating the degree of match. This configuration may include a determination means that, based on the level of the facial feature score calculated by this facial feature comparison means, determines whether the user who performed the drag operation is the person corresponding to the user identification information used during login.

[0025] In this configuration, where facial image data of the user during a drag operation is acquired (case (A)), a facial feature score for the user during the drag operation can be calculated using the acquired facial image data. In this case, the user is simply performing the drag operation to accomplish the intended action, and facial image data for identity verification is acquired without the user being particularly aware of it, thus reducing the effort required of the user.

[0026] <(A) In the case of a combination of facial recognition and (1) the act of affixing an electronic seal> Furthermore, in the case where the configuration is set to acquire facial image data of the user during the drag operation described above, The system includes a seal impression storage means that stores the seal impression data of the electronic seal used by the user, associated with user identification information. The screen display means is, The system can be configured to use drag-and-drop operations to perform the act of applying an electronic seal by the user. This involves placing a seal impression object on the screen and displaying a target area on the screen where the user can drop this seal impression object to complete the application.

[0027] Here, the "target area" is the area on a physical document (on paper) where the seal would be affixed. If there are multiple items to be approved by affixing a seal, multiple target areas (places to drop the seal impression object) may be provided, each corresponding to one of the items. Alternatively, multiple checkboxes or similar selection areas (multiple selection areas) may be provided for each of the items to be approved or not, and only one target area (place to drop the seal impression object) may be used. In the former case, the seal impression object is dragged to each target area corresponding to each item to be approved, resulting in multiple independent drag operations being repeated. That is, the same seal impression object is dragged from the same position (starting position of the drag) to each of the multiple target areas in different locations. On the other hand, in the latter case, the items to be approved are selected using the selection areas such as checkboxes, and then the seal impression object is dragged to a single target area (place to drop the seal impression object), so only one drag operation is required.

[0028] In this configuration, where facial image data of the user is acquired during the drag operation for performing the electronic seal impression (the combination of (A) and (1)), facial image data for identity verification is acquired without the user being particularly aware of it while the seal impression is being performed, thus reducing the effort required of the user. Furthermore, when using a physical seal, the user has to manually move the seal to the area on the paper where it should be stamped, but in this invention, this action is replaced by a drag operation on the screen, making it possible to perform the seal impression and the identity verification for determining whether or not to approve it very naturally.

[0029] <(A) In the case of a combination of facial recognition, (C) authentication based on the pass-through decision result, and (1) the act of affixing an electronic seal> In a configuration where facial image data of the user is acquired during the drag operation for performing the act of affixing an electronic seal as described above, The user information storage means is, In addition to a means of storing information for facial recognition, When dragging a seal impression object, the system is configured to store, as registered personal information, the user's identification information, which includes identification information for a designated pass-through image or drawing target selected from among multiple pass-through image data or multiple pass-through drawing targets, or the pass-through order when passing through multiple designated pass-through areas in sequence, in association with user identification information. The screen display means is, The system is configured to also perform the process of arranging and displaying multiple of the aforementioned passage areas on the screen. The means of obtaining authentication information is, In addition to methods for acquiring facial images, A coordinate acquisition means that performs a process to obtain the coordinate information of the current drag operation instruction position of the dragged seal object, This system includes a passage determination means that uses the coordinate information acquired by this coordinate acquisition means to perform a process to determine whether the drag operation instruction position of the seal impression object has passed through a designated passage area, or a process to determine whether multiple designated passage areas have passed through in the correct order. The means for determining the authentication means is, The system can be configured to use the results of the facial feature score determination (high or low) calculated by the facial feature comparison means and the pass determination results by the pass determination means to perform a process to determine whether the user who performed the drag operation is the person corresponding to the user identification information used during login.

[0030] Here, "coordinate information of the current drag operation instruction position of the dragged seal object" refers to the coordinate information indicating the position of the mouse or touchpad cursor, or the contact position of a finger or other object on the touch panel, during the drag operation of the seal object. The same applies when dragging objects other than seal objects.

[0031] Furthermore, regarding the "process for determining whether or not a designated passage area has been passed through, or the process for determining whether or not multiple designated passage areas have been passed through in the correct order" in the "passage determination means," the former process applies when there is only one designated passage area, and the latter process applies when there are multiple designated passage areas.

[0032] In the former process, displaying multiple pass-through areas on the screen increases the authentication effectiveness, but too many areas mean the user spends a lot of time searching for the designated pass-through area, and it becomes difficult to display all pass-through areas on a single screen. For example, if there are only three pass-through areas, three drag operations will always result in passing through the designated pass-through area, making it preferable to combine it with other authentication methods. On the other hand, if there are 1000 pass-through areas, it becomes extremely difficult for someone other than the legitimate user to pass through all of them, and the probability of accidentally passing through the designated pass-through area also decreases, thus increasing the authentication effectiveness, but it also increases the effort required for the legitimate user to find the designated pass-through area.

[0033] In the latter case, the order of passage is also a factor in the decision, so the number of passage areas can be small. For example, if you prepare passage areas with the numbers 0 through 9, the passage order 1, 2, 3, 4 is different from the passage order 4, 2, 3, 1, so there are 10 to the power of 4 possible passage orders, and the authentication effect is high. This is similar to typical four-digit password authentication.

[0034] In this configuration, where facial image data of the user is acquired during the drag operation to perform the electronic seal impression, and a determination is made as to whether the drag operation instruction position has passed through a designated passage area (in the case of a combination of (A), (C), and (1)), the user must consciously select and pass through a designated passage area from among the multiple passage areas displayed on the screen when performing the drag operation, thus increasing the burden on the user. In other words, the user is burdened in order to perform identity verification. However, since the drag operation itself is performed solely to perform the intended action, the effort required of the user is reduced compared to a case where the user is forced to perform an operation solely for identity verification.

[0035] <(A) Facial recognition, (C) Authentication based on the pass-through decision result, and (1) The act of affixing an electronic seal, and in the case where (C) Authentication utilizes a layered spatial image.> In a configuration where the user's facial image data is acquired during the drag operation to perform the aforementioned electronic seal impression, and a determination is made as to whether or not the drag operation instruction position has passed through the designated passage area, The information storage means for passing through is, The system is configured to store identification information for designated spatial image data used for passage, which is selected in advance by the user from a group of spatial image data in multiple hierarchical levels in which the spatial image data in the upper and lower levels are related to each other. The screen display means is, The system is configured to also perform the process of arranging and displaying on the screen regions formed by each of the spatial image data sets of the highest hierarchical level, as multiple transit regions. The means for determining whether the authentication information acquisition method has passed is: The system can be configured to use coordinate information acquired by a coordinate acquisition means to determine whether the drag operation instruction position of the seal impression object has passed through any of the regions of the spatial image data in the displayed spatial image data group. If it is determined that the object has passed through a region, the display is transitioned from the regions of the higher-level spatial image data group to the regions of the lower-level spatial image data group. During this transition, the system can also perform a process to determine whether the drag operation instruction position of the dragged seal impression object has passed through the region of the specified spatial image data.

[0036] In this configuration, where the user's facial image data is acquired during a drag operation to perform the electronic seal impression, and it is determined whether the drag operation instruction position has passed through the area of ​​the specified spatial image data (a combination of (A), (C), and (1), and where (C) utilizes hierarchical spatial images), the user can reach the area of ​​the specified spatial image data by navigating the display of each area of ​​the hierarchical spatial image data from the higher hierarchical level to the lower hierarchical level. In this case, since the spatial image data of the upper and lower hierarchical levels are related in content (there is a content-based inclusion relationship), the user can reach the area of ​​the specified spatial image data by performing the drag operation while considering this content-based relationship. Therefore, although there is only one specified spatial image data to pass through, the user's selection by drag operation is made among the many hierarchical spatial image data areas that are displayed, thus increasing the authentication effect. In other words, as mentioned above, if there is only one designated passage area, the order of passage cannot be used as a determining factor. Therefore, the authentication effect cannot be improved unless the number of passage areas (number of options) displayed on the screen is increased. However, if the number of passage areas is increased too much, it will take the user time to find the designated passage area, and it will become difficult to display all passage areas on a single screen. However, these problems can be solved by using hierarchical spatial image data.

[0037] <(A) A combination of facial recognition, (C) authentication based on the pass / fail judgment result, and (1) the act of affixing an electronic seal, and (A) when the average facial feature score is calculated using facial recognition.> In a configuration where multiple facial image data of a user during a drag operation to perform the aforementioned electronic seal impression are acquired, an average facial feature score is calculated, and it is determined whether or not the drag operation instruction position has passed through a designated passage area, In a configuration where the user's facial image data is acquired during the drag operation to perform the aforementioned electronic seal impression, and a determination is made as to whether or not the drag operation instruction position has passed through the designated passage area, The means of acquiring facial images is, The system is configured to repeatedly execute the process of acquiring facial image data during a single drag operation. The facial feature extraction method of the authentication method is The system is configured to perform a process of extracting the facial features of the user during a drag operation from each of multiple facial image data acquired by the facial image acquisition means, or, in addition to this process, to extract the facial features of the user from the user's facial image data if the user's facial image data is stored as registered user information in the facial recognition information storage means. The facial feature comparison method of the authentication method is, The system is configured to perform a process in which each of the multiple facial features of the user during a drag operation, extracted by the facial feature extraction means, is compared with the user's facial features stored as registered user information in the facial recognition information storage means, or with the user's facial features extracted by the facial feature extraction means from the user's facial image data stored as registered user information in the facial recognition information storage means, calculates multiple facial feature scores indicating the degree of match, and calculates an average facial feature score by averaging these multiple facial feature scores. The means for determining the authentication means is, The system can be configured to use the results of the judgment of whether the average facial feature score calculated by the facial feature comparison means is high or low, and the results of the pass judgment means, to perform a process to determine whether the user who performed the drag operation is the person corresponding to the user identification information used during login.

[0038] In this configuration, multiple facial image data of the user during the drag operation to perform the electronic seal impression are acquired, an average facial feature score is calculated, and it is determined whether or not the drag operation instruction position has passed through a designated passage area. Since the determination is made using the average facial feature score during the drag operation, stable user authentication can be achieved.

[0039] <(1) In the case of affixing an electronic seal> As explained above in <Arbitrary nature of the combination of purpose and authentication information>, (1) the act of affixing an electronic seal does not necessarily require (A) facial recognition, and can be combined with any of the authentication methods, including (A), (B), and (C).

[0040] <(2) In the case of answering a question> Similarly, the act of answering the questions (2) can be combined with any of the authentication methods, including the authentication methods (A), (B), and (C), so the following configuration is possible.

[0041] In other words, in the aforementioned information processing system, It is equipped with a question storage means for storing question data for language learning or other learning purposes, or for language proficiency tests or other examinations, which are presented to the user. The screen display means is, The system can be configured to use problem data stored in a problem memory means to place answer objects representing words or other answer elements on the screen as objects used for drag-and-drop operations for the user to perform answering actions for word rearrangement problems, fill-in-the-blank problems, or other problems, and to display a target area on the screen for dropping these answer objects to complete the answering action.

[0042] In this configuration, where the intended action is the act of answering a problem, various authentication information for user verification is acquired during the drag operation to answer the problem. In this case, since the user is only dragging the answer object to answer the problem, the user effort can be reduced compared to cases where user operation is required solely for user verification.

[0043] (3) In the case of an execution action that results in the transfer of assets. Furthermore, (3) the execution act that causes the transfer of assets can be combined with any of the authentications of the person(s), including the authentications of (A), (B), and (C), so the following configuration is possible.

[0044] In other words, in the aforementioned information processing system, The screen display means is, The system can be configured to use drag-and-drop operations to perform execution actions that result in the transfer of funds to a financial institution account, transfers between accounts, buying and selling of securities or other financial products, e-commerce, in-game charges, or other asset transfers. These execution objects may be placed on the screen to represent cash, virtual currency, financial products, or other financial assets, the source account or owner of the financial assets, the object of an e-commerce transaction, a game character, or other representations related to asset transfers. The system can also be configured to display a target area on the screen where the execution object can be dropped to complete the execution action.

[0045] In this configuration, where the intended action is an execution action that results in the transfer of assets, various authentication information for identity verification is acquired during the drag operation required to perform the execution action that results in the transfer of assets. In this case, since the user is only dragging the execution object to execute an action that involves the transfer of assets, the user's effort can be reduced compared to cases where user operation is required solely for identity verification.

[0046] <(C) In the case of certification based on the pass / fail judgment result> As explained above in <Arbitrary nature of the combination of purpose act and authentication information>, (C) authentication based on the pass judgment result does not necessarily have to be a combination of (A) facial recognition and (1) the act of affixing an electronic seal, and may be combined with any personal authentication method, including (A) facial recognition and (B) voiceprint recognition, and can be combined with various purpose acts, including (1) the act of affixing an electronic seal, (2) the act of answering a question, and (3) the execution act that results in the transfer of assets.

[0047] <(B) In the case of voiceprint authentication> As explained above in <Arbitrary nature of the combination of purpose act and authentication information>, (B) voiceprint authentication may be combined with any form of personal authentication, including (A) facial recognition and (C) authentication based on the pass / fail judgment result. Furthermore, it can be combined with various purpose acts, including (1) the act of affixing an electronic seal, (2) the act of answering a question, and (3) the execution act that results in the transfer of assets. Therefore, the following configuration is possible.

[0048] In other words, in the aforementioned information processing system, The user information storage means is, The system includes a voiceprint storage means that stores the registered individual's voiceprint data in association with user identification information. The means of obtaining authentication information is, It is configured to include a voice acquisition means that performs a process to acquire voice data of the user during a drag operation using a microphone, The authentication method is, A voiceprint analysis means performs voiceprint analysis using the user's voice data acquired by the voice acquisition means during a drag operation, and executes a process to create voiceprint data for voiceprint authentication. A voiceprint comparison means performs a process to compare the voiceprint data of the user during the drag operation obtained by this voiceprint analysis means with the voiceprint data of the same person stored as registered personal information in the voiceprint storage means, and calculates a voiceprint score indicating the degree of match. This configuration may include a determination means that performs a process to determine whether the user who performed the drag operation is the person corresponding to the user identification information used during login, based on the level of the voiceprint score calculated by this voiceprint comparison means.

[0049] Here, "user voice data during drag operation" may be voice data that is necessarily uttered in order to perform the intended action (for example, when asked to speak due to a language problem, or when performing a drag operation to perform the intended action while on the phone), or it may be voice data that is deliberately uttered while performing a drag operation for the purpose of voiceprint authentication.

[0050] In this configuration, where voice data is acquired from the user during a drag operation for voiceprint authentication, the acquisition of voice data for voiceprint authentication occurs simultaneously with the drag operation to perform the intended action. This reduces the user effort compared to cases where a separate user operation is required solely for voiceprint authentication. It should be noted that even if the user intentionally speaks for voiceprint authentication, no manual user operation is required; the drag operation is performed solely to perform the intended action.

[0051] Furthermore, in cases where vocalization occurs not intentionally for voiceprint authentication, but inevitably as a result of performing the intended action, such as when answering questions for language learning or language proficiency tests, the structure is as follows.

[0052] <(B) A combination of voiceprint authentication and (2) the act of answering a question, and where (2) the question is a language learning or language proficiency test question that requires the user to speak.> In other words, in a configuration where voice data of the user during the drag operation described above is acquired and voiceprint authentication is performed, It includes a question storage means for storing question data for language learning or language proficiency tests to be presented to the user, The screen display means is, The system is configured to use problem data stored in a problem memory means to place answer objects representing words or other answer elements on the screen as drag-and-drop objects used for the user to perform a verbal answer action for word rearrangement problems, fill-in-the-blank problems, or other language-related problems, and to display a target area on the screen for dropping these answer objects to complete the answer action. The voice acquisition means for the authentication information acquisition means is, The system can be configured to use a microphone to capture audio data when a user speaks while performing a drag operation and provides an answer.

[0053] In this configuration, where voice data spoken by the user during a drag operation to answer a question for language learning or language proficiency testing is acquired and voiceprint authentication is performed (a combination of (B) and (2), and the question in (2) is a question for language learning or language proficiency testing that requires the user to speak), the user will inevitably speak when answering the question, thus further reducing the effort required of the user.

[0054] <Inventive Program> Furthermore, the program of the present invention is intended to enable a computer to function as the information processing system described above.

[0055] Furthermore, the above program or any part thereof can be recorded and stored on various recording media such as compact discs (CDs), digital versatile discs (DVDs), USB memory sticks, hard disk drives (HDDs), and solid-state drives (SSDs) for distribution, etc. It can also be transmitted using transmission media such as wired networks like local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), the internet, intranets, extranets, etc., or wireless communication networks, or combinations thereof, and can also be transmitted on carrier waves. Moreover, the above program may be part of another program, or may be recorded on a recording medium together with a separate program. [Effects of the Invention]

[0056] As described above, according to the present invention, when a user performs a drag operation to carry out a desired action, authentication information is acquired during that drag operation. This reduces the effort required of the user compared to cases where user operation is required solely for authentication purposes. Furthermore, instead of performing authentication uniformly for all actions, as is done when logging into a system, authentication can be performed only for necessary actions. In addition, it has the effect of preventing fraud due to changes in the operator after login. [Brief explanation of the drawing]

[0057] [Figure 1] An overall configuration diagram of an electronic seal system, which is an information processing system according to the first embodiment of the present invention. [Figure 2] A flowchart illustrating the flow of the user authentication process using the electronic seal system of the first embodiment. [Figure 3] An explanatory diagram of the act of stamping on the work screen in the electronic seal system of the first embodiment. [Figure 4] Another explanatory diagram of the act of stamping on the work screen in the electronic seal system of the first embodiment. [Figure 5] A first explanatory diagram showing the manner of passage through the designated passage area in the electronic seal system of the first embodiment. [Figure 6] A second explanatory diagram showing the manner of passage through the designated passage area in the electronic seal system of the first embodiment. [Figure 7] A third explanatory diagram showing the manner of passage through the designated passage area in the electronic seal system of the first embodiment. [Figure 8] A fourth explanatory diagram showing the manner of passage through the designated passage area in the electronic seal system of the first embodiment. [Figure 9] A flowchart illustrating the flow of the process for determining whether a specified spatial image data region has been passed through in the electronic seal system of the first embodiment. [Figure 10] A flowchart illustrating the flow of the process for determining whether an area is a designated pass-through image data or a designated pass-through drawing target area in the electronic seal system of the first embodiment. [Figure 11] An overall configuration diagram of a problem system, which is an information processing system according to a second embodiment of the present invention. [Figure 12] A flowchart illustrating the flow of the user authentication process using the problem system of the second embodiment. [Figure 13] A diagram illustrating the answering process on the problem screen in the problem system of the second embodiment. [Figure 14] An overall configuration diagram of an asset transfer system, which is an information processing system according to the third embodiment of the present invention. [Figure 15] A flowchart illustrating the flow of the user authentication process using the asset transfer system of the third embodiment. [Figure 16] An explanatory diagram of a transfer operation, which is an execution action on the transfer screen in the asset transfer system of the third embodiment. [Figure 17] An explanatory diagram of the product purchase action, which is an execution action on the product purchase screen in the asset transfer system of the third embodiment. [Modes for carrying out the invention]

[0058] Each embodiment of the present invention will be described below with reference to the drawings.

[0059] [First Embodiment]

[0060] Figure 1 shows the overall configuration of the electronic seal system 10, which is an information processing system according to the first embodiment of the present invention, and Figure 2 shows the flow of the personal authentication process as a flowchart. Figures 3 and 4 are explanatory diagrams of the act of stamping on the work screen, and Figures 5 to 8 are explanatory diagrams of the manner of passing through the designated passage area. Furthermore, Figure 9 shows the flow of the process for determining whether to pass through the area of ​​the designated spatial image data as a flowchart, and Figure 10 shows the flow of the process for determining whether to pass through the area of ​​the designated passage image data or the area to be drawn as a designated passage as a flowchart.

[0061] <Overall configuration of the electronic seal system 10>

[0062] In Figure 1, the electronic seal system 10 is configured to include multiple work servers 20 (20A, 20B, 20C, ...), an authentication server 30, an electronic seal server 50, and multiple (usually many) user terminals 60.

[0063] Each work server 20 and each user terminal 60 are connected via network 1. Network 1 is, for example, an external network such as the internet. Each work server 20, the authentication server 30, and the electronic seal server 50 are connected via network 2. Network 2 is, for example, an internal network such as a LAN or intranet, but like network 1, it may also be an external network such as the internet. If network 2 is an external network (shared with network 1), the operating and management entities of each work server 20, authentication server 30, and electronic seal server 50 may be different businesses.

[0064] Furthermore, in this first embodiment, each work server 20, authentication server 30, and electronic seal server 50 are configured by separate computers, but they may also be configured by a single computer.

[0065] Each work server 20 performs processing related to various tasks that require the application of an electronic seal (e.g., contracts, agreements, approvals, confirmations, etc.), and each is composed of one or more computers. In other words, one work server 20 may be composed of one computer, or it may be composed of multiple computers, such as a web server, web application server, database server, etc.

[0066] Each work server 20 is equipped with a screen display means 21. The first work server 20A, the second work server 20B, the third work server 20C, ... handle different types of work, so the screen display means 21 of each work server 20 executes the process of displaying different work screens. Although not shown in the diagram, each work server 20 also stores various data necessary for screen display. This screen display means 21 is implemented by a central processing unit (CPU) located inside each work server 20, and one or more programs that define the operating procedure of this CPU. Details of the screen display means 21 will be described later.

[0067] The authentication server 30 is composed of one or more computers and includes processing means 30A that perform various processes related to user authentication, and storage means 40 that store various data necessary for the execution of the various processes by the processing means 30A.

[0068] The processing means 30A includes a user information registration means 31 and an authentication means 32. The authentication means 32 includes a basic authentication means 32A, a face feature extraction means 32B, a face feature comparison means 32C, a voiceprint analysis means 32D, a voiceprint comparison means 32E, a pass-through determination assistance means 32F, and a determination means 32G.

[0069] Each of the means 31, 32 (32A to 32G) included in the processing means 30A is implemented by a central processing unit (CPU) located inside the authentication server 30, and one or more programs that define the operating procedures of this CPU. Details of each of the means 31, 32 (32A to 32G) will be described later.

[0070] The storage means 40 includes a user information storage means 41, a spatial image storage means 42, and a pass-through image storage means 43. The user information storage means 41 includes a basic information storage means 41A, a facial recognition information storage means 41B, a pass-through information storage means 41C, and a voiceprint storage means 41D.

[0071] The individual storage devices 41(41A~41D), 42, and 43 included in the storage device 40 can be, for example, hard disk drives (HDDs), solid-state drives (SSDs), etc. Details of each storage device 41(41A~41D), 42, and 43 will be described later.

[0072] The electronic seal server 50 is composed of one or more computers and includes a registration / management means 51 that performs processing related to the registration and management of electronic seals, and a seal impression storage means 52 that stores seal impression data. The registration / management means 51 is implemented by a central processing unit (CPU) located inside the electronic seal server 50, and one or more programs that define the operating procedures of this CPU. Details of the registration / management means 51 will be described later. In addition, the seal impression storage means 52 can be, for example, a hard disk drive (HDD), a solid state drive (SSD), etc. Details of the seal impression storage means 52 will be described later.

[0073] The user terminal 60 is a computer and includes a display means 70 such as an LCD display, an input means 71 such as a mouse, keyboard, touchpad, or touch panel, a camera 72, and a microphone 73.

[0074] The user terminal 60 is configured to include processing means 60A. Although not shown in the illustration, the user terminal 60 also stores data necessary for the execution of processing by the processing means 60A. The processing means 60A is configured to include a drag operation determination means 61, an authentication information acquisition means 62, and a drop permission determination means 63. The authentication information acquisition means 62 is configured to include a face image acquisition means 62A, a coordinate acquisition means 62B, a pass-through determination means 62C, and an audio acquisition means 62D.

[0075] Each of the means 61, 62 (62A to 62D) included in the processing means 60A is implemented by a central processing unit (CPU) provided in the user terminal 60, and one or more programs that define the operating procedures of this CPU. This program may be, for example, a program provided in conjunction with a web page (e.g., JavaScript (ECMAScript), etc.: JavaScript is a registered trademark), or a program that is pre-installed on the user terminal 60.

[0076] <Detailed configuration of working server 20>

[0077] The screen display means 21 receives a request to display a work screen transmitted from the user terminal 60 via the network 1, creates display data for various work screens as shown in the work screens 100 and 110 in Figures 3 and 4, and executes the process of transmitting it to the user terminal 60 via the network 1. As a result, the various work screens are displayed on the display means 70 of the user terminal 60.

[0078] In Figure 3, the work screen 100 includes a document display unit 101 that displays the electronic document to be stamped (in the example in Figure 3, a project proposal), a stamp impression object 102 that can be dragged, a target area 103 where the stamp impression object 102 is dropped to complete the stamping action, and a pass-through area arrangement unit 104 that displays and arranges multiple pass-through areas (including designated pass-through areas) through which the stamp impression object 102 (more precisely, the drag operation instruction position for the stamp impression object 102) passes for user authentication. The target area 103 is displayed within the document display unit 101. In the example in Figure 3, Kukan Taro, who created the project proposal, drags the stamp impression object 102 to apply his electronic seal, passes it through the pass-through area arrangement unit 104, and drops it into the target area 103. If the head of the planning department approves Kukan Taro's proposal, the head of the planning department logs in, drags the stamp impression object of the head of the department's own electronic seal, and drops it into the target area indicating the approval stamp.

[0079] In Figure 4, the work screen 110 includes a document display unit 111 that displays the electronic document to be stamped (in the example in Figure 4, a confirmation sheet), a stamp impression object 102 (same as in Figure 3), a target area 113 where the stamp impression object 102 is dropped to complete the stamping process, and a pass-through area placement unit 104 (same as in Figure 3). The target area 113 is displayed within the document display unit 111. In the example in Figure 4, Taro Kukan selects the confirmed items using checkboxes, then drags the stamp impression object 102, passes it through the pass-through area placement unit 104, and drops it into the target area 113. Therefore, although there are multiple check items, the drag-and-drop operation only needs to be performed once. Alternatively, a target area can be provided for each of the multiple check items. In that case, the stamp impression object 102 would be dragged to each target area corresponding to the confirmed items, so the drag-and-drop operation would be repeated for each confirmed item.

[0080] Here, the drag operation instruction position is the position for displaying the imprint object 102 while it is being dragged. If the input means 71 of the user terminal 60 is a mouse or touchpad, it is the position of the cursor; if it is a touch panel, it is the contact position of a finger or the like.

[0081] Furthermore, the data (file) of the electronic document to be stamped, displayed on the document to be stamped display units 101 and 111, may be stored and managed on the work server 20, or it may be stored on the user terminal 60. In the latter case, the user performs a read operation of the data of the electronic document to be stamped on the user terminal 60.

[0082] The screen display means 21 accesses the electronic seal server 50 to create display data for the seal impression object 102 and retrieves the user's seal impression data stored in the seal impression storage means 62. At this time, the screen display means 21 sends the user ID (user identification information) entered by the user when logging into the work server 20 to the electronic seal server 50 and receives the seal impression data stored in association with the user ID from the electronic seal registration and management means 51 of the electronic seal server 50.

[0083] Furthermore, the screen display means 21 accesses the authentication server 30 to create display data for the passage area arrangement unit 104 and obtains either a hierarchical group of spatial image data stored in the spatial image storage means 42 (each hierarchical group of spatial image data includes one specified spatial image data selected by the user), or multiple passage image data stored in the passage image storage means 43 (which includes two or more specified passage image data selected by the user). Note that if the passage area (including the specified passage area) is to be used as the area for passage drawing targets (including the specified passage drawing targets) rather than as the area for passage image data (including the specified passage image data) (for example, when drawing numbers from 0 to 9, alphabet letters A, B, C, ..., Greek letters α, β, γ, ..., or Chinese characters such as East, West, North, South, Winter, Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune, etc.), then passage image data is not necessary. This is because no display processing using image data is performed.

[0084] In this case, as shown in the work screens 100 and 110 in Figures 3 and 4, hierarchical spatial image data is used for user authentication, so the screen display means 21 initially acquires the spatial image data set of the highest hierarchical level. Note that, depending on how the user specifies, the spatial image data set of the highest hierarchical level usually does not include the specified spatial image data to be passed through. This is because such a specification would result in insufficient user authentication effectiveness. Subsequently, the screen display means 21 responds to a transmission request from the user terminal 60's pass-through determination means 62C, accesses the authentication server 30, acquires the spatial image data set of the lower hierarchical level (the hierarchical level related to the transmission request from the pass-through determination means 62C) stored in the spatial image storage means 42, and transmits the acquired spatial image data set to the user terminal 60.

[0085] Furthermore, when using transit image data other than hierarchical spatial image data for user authentication, that is, when using the transit order of the designated transit image data area for user authentication (see Figure 5), the screen display means 21 acquires all of the multiple transit image data (including two or more designated transit image data to be passed through) from the transit image storage means 43 to be placed and displayed in the transit area arrangement section. Note that when using transit drawing targets for user authentication (for example, when drawing numbers from 0 to 9, etc.), that is, when using the transit order of the designated transit drawing target area for user authentication (see Figures 6 to 8), no display processing using image data is performed, so acquiring image data is not necessary. However, numbers from 0 to 9, etc., may be prepared as image data, in which case multiple transit image data will be acquired from the transit image storage means 43.

[0086] Furthermore, the screen display means 21 accesses the authentication server 30 to obtain information to be passed to the user terminal 60's passage determination means 62C, and obtains identification information of a designated spatial image data stored in the passage information storage means 41C (in the case of Figures 3 and 4), or information on the passage order of the designated passage image data or the area to be drawn for passage (in the case of Figures 5 to 8). Here, the identification information of one designated spatial image data to be passed (in the case of Figures 3 and 4) is stored in the passage information storage means 41C in association with the user ID (user identification information) entered at the time of login, so the screen display means 21 obtains that information from the passage information storage means 41C using the user ID. Alternatively, the identification information of each of the two or more designated pass-through image data to be passed and their order (information indicating which pass-through image data should be passed and in what order) (in the case of Figure 5), or the identification information of each of the two or more designated pass-through drawing targets to be passed and their order (information indicating which pass-through drawing targets should be passed and in what order) (in the cases of Figures 6 to 8) are stored in the pass-through information storage means 41C in association with the user ID (user identification information) entered at the time of login, so the screen display means 21 retrieves this information from the pass-through information storage means 41C using the user ID.

[0087] Furthermore, the screen display means 21 of the work server 20 plays the role of relaying information between the user terminal 60 and the authentication server 30 in order to perform user authentication in this invention. In this case, if the operating and management entities (businesses) of each work server 20 and the authentication server 30 are different, or if the operating and management entities (businesses) of each work server 20 are different, a mechanism such as OAuth authentication may be used when obtaining various information from the authentication server 30. However, for the sake of explanation, this explanation will be based on the premise that each work server 20 is not malicious, or that the operating and management entities of each work server 20 and the authentication server 30 are the same. The same applies to other embodiments.

[0088] <Detailed configuration of authentication server 30>

[0089] The user information registration means 31 receives input of basic information (such as a password for a user ID) transmitted from the user terminal 60 via one of the work servers 20, and stores the received basic information in the basic information storage means 41A in association with the user ID (user identification information). It also receives input of pre-registration personal information (information used for personal authentication in the present invention) transmitted from the user terminal 60 via one of the work servers 20, and stores the received registered personal information in the facial recognition information storage means 41B, the passage information storage means 41C, and the voiceprint storage means 41D in association with the user ID (user identification information).

[0090] Here, the registered personal information includes the user's face image data, or the user's face features extracted from that face image data by the face feature extraction means 32B, and these are stored in the face recognition information storage means 41B.

[0091] Furthermore, the registered personal information includes information about a designated passage area (an area selected by the user from among multiple passage areas), which is stored in the passage information storage means 41C. When performing personal authentication using hierarchical spatial image data, the information about the designated passage area stored in the passage information storage means 41C becomes the identification information for the designated spatial image data (a single spatial image data selected by the user from among multiple spatial image data). This is because there is only one area of ​​the designated spatial image data to be passed through. Furthermore, when performing user authentication using pass-through image data other than hierarchical spatial image data, or pass-through drawing targets (information such as numbers or characters, rather than image data), the information of the designated pass-through area stored in the pass-through information storage means 41C will be information on the pass-through order of the areas of the designated pass-through image data (two or more pass-through image data selected by the user from among multiple pass-through image data) (information indicating the 1st, 2nd, ... designated pass-through image data = each identification information and their order), or information on the pass-through order of the areas of the designated pass-through drawing targets (two or more pass-through drawing targets selected by the user from among multiple pass-through drawing targets) (information indicating the 1st, 2nd, ... designated pass-through drawing targets = each identification information and their order).

[0092] Furthermore, the registered personal information includes voiceprint data obtained by voiceprint analysis of the user's own voice data using the voiceprint analysis means 32D, which is stored in the voiceprint storage means 41D.

[0093] The authentication means 32 uses various information stored in the user information storage means 41 and other components included in the storage means 40 to perform various processes for authenticating the user.

[0094] The basic authentication means 32A, upon login to the work server 20, compares the password entered along with the user ID (user identification information) at the user terminal 60 with the password corresponding to the user ID stored in the basic information storage means 41A, and performs user authentication processing based on whether they match. This user authentication at login is a conventional basic authentication and differs from the user authentication of the present invention, which is performed using authentication information acquired during a drag operation. Therefore, in this first embodiment, the user authentication of the present invention is part of a multi-stage authentication performed in addition to the basic authentication described above. Note that the basic authentication may be omitted, and only the user authentication of the present invention may be performed.

[0095] The facial feature extraction means 32B receives facial image data acquired by the facial image acquisition means 62A of the user terminal 60 and transmitted via the network 1 and the work server 20, and performs a process to extract the facial features of the user during the drag operation from this facial image data. Furthermore, if the facial recognition information storage means 41B stores the user's facial image data, rather than the user's facial features, as registered user information, the facial feature extraction means 32B also performs a process to extract the user's facial features from this user's facial image data.

[0096] Furthermore, if the face image acquisition means 62A of the user terminal 60 repeatedly acquires face image data during a single drag operation, the face feature extraction means 32B executes a process to extract the face features of the user during the drag operation from each of the multiple face image data acquired by the face image acquisition means 62A.

[0097] The facial feature comparison means 32C compares the facial features of the user during the drag operation, extracted by the facial feature extraction means 32B, with the user's facial features stored in the facial recognition information storage means 41B as registered user information (facial features stored in association with the user ID (user identification information) entered at login), and calculates a facial feature score indicating the degree of match. Furthermore, if the facial recognition information storage means 41B stores the user's facial image data instead of their facial features as registered user information, the facial feature comparison means 32C compares the facial features of the user during the drag operation, extracted by the facial feature extraction means 32B, with the user's facial image data (facial image data stored in association with the user ID (user identification information) entered at login) stored in the facial recognition information storage means 41B as registered user information, and calculates a facial feature score indicating the degree of match.

[0098] Furthermore, if the face image acquisition means 62A of the user terminal 60 repeatedly acquires face image data during a single drag operation, the face feature comparison means 32C compares each of the multiple face features of the user during the drag operation, extracted by the face feature extraction means 32B, with the user's face features stored as registered user information in the face authentication information storage means 41B, calculates multiple face feature scores indicating the degree of match, and calculates an average face feature score by averaging these multiple face feature scores. Furthermore, if the facial recognition information storage means 41B stores the user's facial image data, rather than the user's facial features, as registered user information, the facial feature comparison means 32C compares each of the multiple facial features of the user during the drag operation, extracted by the facial feature extraction means 32B, with the user's facial features extracted by the facial feature extraction means 32B from the user's facial image data stored in the facial recognition information storage means 41B as registered user information. The face feature comparison means 32C calculates multiple facial feature scores indicating the degree of agreement and averages these multiple facial feature scores to calculate an average facial feature score.

[0099] The voiceprint analysis means 32D receives the user's voice data during a drag operation, which is acquired by the voice acquisition means 62D of the user terminal 60 and transmitted via the network 1 and the work server 20. It then performs voiceprint analysis using this voice data and executes a process to create voiceprint data for voiceprint authentication.

[0100] The voiceprint comparison means 32E compares the voiceprint data of the user during the drag operation obtained by the voiceprint analysis means 32D with the user's own voiceprint data stored in the voiceprint storage means 41D as registered user information (voiceprint data stored in association with the user ID (user identification information) entered at the time of login), and performs a process to calculate a voiceprint score indicating the degree of match.

[0101] The passage determination assisting means 32F performs processing to assist the passage determination means 62C of the user terminal 60. Therefore, in this first embodiment of the present invention, the passage determination means is realized by the passage determination means 62C of the user terminal 60 and this passage determination assisting means 32F.

[0102] Specifically, the passage determination assistance means 32F receives the passage determination result transmitted from the passage determination means 62C of the user terminal 60 via the network 1 and the work server 20. The passage determination assistance means 32F also receives a spatial image update request (a request to transmit a group of lower-level spatial image data) transmitted from the passage determination means 62C of the user terminal 60 via the network 1 and the work server 20, obtains the lower-level spatial image data related to this request from the spatial image storage means 42, and transmits it to the passage determination means 62C of the user terminal 60 via the work server 20 and the network 1.

[0103] The determination means 32G uses information obtained from at least one of the facial feature comparison means 32C, voiceprint comparison means 32E, and passage determination assistance means 32F to perform an authentication process to determine whether the user who performed the drag operation is the person corresponding to the user ID (user identification information) used at login, and to transmit the authentication result obtained from this process to the user terminal 60 via the work server 20 and network 1.

[0104] In this first embodiment, the authentication information acquired during the drag operation includes (A) facial image data used for facial recognition, (B) voice data used for voiceprint recognition, and (C) the pass judgment result. Authentication may be performed using all of this information, or, at the discretion of the user or the operator of the work server 20, authentication may be performed using only some of the information. Furthermore, if (A) facial image data used for facial recognition cannot be acquired due to a malfunction of the camera 72 or the like, some of the information may be used as a supplement, such as by acquiring (B) voice data used for voiceprint recognition.

[0105] (A) When only facial recognition is performed, the determination means 32G determines whether or not the person is the person in question based on the level of the facial feature score calculated by the facial feature comparison means 32C. That is, a threshold for the facial feature score is predetermined, and the determination means 32G determines whether the calculated facial feature score is equal to or greater than that threshold. Furthermore, when an average facial feature score is calculated from multiple facial image data, a threshold for the average facial feature score is predetermined, and the determination means 32G determines whether the calculated average facial feature score is equal to or greater than that threshold.

[0106] (B) When only voiceprint authentication is performed, the determination means 32G determines whether or not the person is the person in question based on the level of the voiceprint score calculated by the voiceprint comparison means 32E. That is, a threshold for the voiceprint score is predetermined, and the determination means 32G determines whether or not the calculated voiceprint score is equal to or exceeds that threshold.

[0107] (C) If authentication is performed solely based on the pass-through determination result, the determination means 32G determines whether the person is the person in question based on the pass-through determination result, which is whether or not the person has passed through the designated pass-through area, or whether or not the person has passed through multiple designated pass-through areas in the correct order. Note that if authentication is not performed based on the pass-through determination result, the pass-through area arrangement section 104 shown in Figures 3 and 4 will not be displayed.

[0108] When (A) facial recognition and (B) voiceprint recognition are combined, the determination means 32G determines whether or not the person is the person in question based on the level of the facial feature score (or average facial feature score) calculated by the facial feature comparison means 32C and the level of the voiceprint score calculated by the voiceprint comparison means 32E. That is, using the threshold judgment result of the facial feature score (or average facial feature score) and the threshold judgment result of the voiceprint score, if both threshold judgment results are high, it is determined that the person is the person in question, and if either threshold judgment result is low, it is determined that the person is not the person in question. However, if either threshold judgment result is high, it may be determined that the person is the person in question. Such a judgment may be made, for example, when some authentication information is unavailable or deemed unreliable due to a malfunction of the camera 72 or microphone 73.

[0109] Furthermore, when (A) facial recognition and (B) voiceprint recognition are combined, the determination means 32G may integrate the facial feature score (or average facial feature score) and the voiceprint score to calculate the degree of identity, and determine whether or not the person is the person based on the level of the calculated degree of identity. That is, a threshold for the degree of identity is predetermined, and it is determined whether or not the calculated degree of identity is equal to or exceeds that threshold. The degree of identity can be, for example, a simple average or a weighted average of the facial feature score (or average facial feature score) and the voiceprint score. For a weighted average, for example, if facial information is to be emphasized, the ratio (ratio of coefficients) of the facial feature score (or average facial feature score) and the voiceprint score can be set to 2:1 or 3:1, etc.

[0110] When (A) facial recognition and (C) authentication based on the pass-through decision result are combined, the determination means 32G uses the threshold determination result of the high or low facial feature score (or average facial feature score) calculated by the facial feature comparison means 32C, and the pass-through decision result of the user terminal 60's pass-through decision means 62C to determine whether or not the person is the person in question. In this case as well, if both determination results are favorable, the person is determined to be the person in question, and if either determination result is unfavorable, the person is determined to be the person in question. Furthermore, if either determination result is favorable, the person may be determined to be the person in question. Such a determination may be made, for example, when some authentication information is unavailable or deemed unreliable due to a malfunction of the camera 72, etc.

[0111] (B) When combining voiceprint authentication and (C) authentication based on the pass judgment result, the determination means 32G uses the threshold judgment result of the voiceprint score calculated by the voiceprint comparison means 32E and the pass judgment result of the user terminal 60's pass judgment means 62C to determine whether or not the person is the real person. In this case as well, if both judgment results are favorable, the person is determined to be the real person, and if either judgment result is unfavorable, the person is determined to be the real person. Furthermore, if either judgment result is favorable, the person may be determined to be the real person. Such a judgment may be made, for example, when some authentication information is unavailable or deemed unreliable due to a malfunction of the microphone 73, etc.

[0112] When combining (A) facial recognition, (B) voiceprint recognition, and (C) authentication based on the pass-through decision result, the determination means 32G uses the threshold judgment result of the high or low facial feature score (or average facial feature score) calculated by the facial feature comparison means 32C, the threshold judgment result of the high or low voiceprint score calculated by the voiceprint comparison means 32E, and the pass-through decision result of the pass-through decision means 62C of the user terminal 60 to determine whether or not the person is the person in question. In this case, if all judgment results are favorable, the person is determined to be the person in question, and if any of the judgment results are unfavorable, the person is determined to be the person in question. Alternatively, if any of the judgment results are favorable, the person may be determined to be the person in question, or if two or more judgment results are favorable. Examples of situations in which such a judgment may be made include cases where some authentication information is unavailable or deemed unreliable due to a malfunction of the camera 72 or microphone 73, etc.

[0113] Furthermore, when combining (A) facial recognition, (B) voiceprint recognition, and (C) authentication based on the pass-through decision result, the determination means 32G may determine whether or not the person is the real person using the threshold determination result of the combined score of the voiceprint score and the facial feature score (or average facial feature score), and the pass-through decision result of the pass-through decision means 62C of the user terminal 60. In this case, if both determination results are favorable, the person is determined to be the real person, and if either determination result is unfavorable, the person is determined to be the real person. Alternatively, if either determination result is favorable, the person may be determined to be the real person. Examples of situations in which such a determination may be made include cases where some authentication information is unavailable or deemed unreliable due to a malfunction of the camera 72 or microphone 73, etc.

[0114] The user information storage means 41 stores registered personal information for user authentication, which has been pre-registered by the user, in association with the user ID (user identification information).

[0115] The basic information storage means 41A stores basic information, including conventional basic authentication information such as passwords, in association with a user ID (user identification information).

[0116] The facial recognition information storage means 41B stores, as registered personal information, the user's facial image data or the user's facial features extracted from this facial image data, in association with the user ID (user identification information).

[0117] The transit information storage means 41C stores information about the designated transit area as registered user information, associated with the user ID (user identification information). This information about the designated transit area is either identification information of the designated spatial image data, or information about the transit order of the designated transit image data or the region to be drawn for transit (each identification piece and their order).

[0118] The voiceprint storage means 41D stores the voiceprint data of the registered individual as registered individual information, associating it with the user ID (user identification information).

[0119] The spatial image storage means 42 stores multiple hierarchical spatial image data, that is, multiple spatial image data and hierarchical information that defines the hierarchical relationships (subordination relationships) of each spatial image data.

[0120] The passing image storage means 43 stores multiple passing image data. Although spatial image data is also a type of passing image data, the passing image data stored in this passing image storage means 43 is passing image data other than spatial image data. The passing image data can be images of any kind, for example, images of nature such as the sea, mountains, and rivers, images of hobbies such as tennis, skiing, and baseball, or images of celebrities. When a user selects images of cherry blossoms, cats, and trains from these images and specifies them in that order, the cherry blossom image becomes the first specified passing image data, the cat image becomes the second specified passing image data, and the train image becomes the third specified passing image data.

[0121] <Detailed configuration of the electronic seal server 50>

[0122] The electronic seal registration and management means 51 receives user seal impression data that has been entered or newly created on the user terminal 60 and transmitted via network 1 and one of the work servers 20, and performs a registration process to store the received seal impression data in the seal impression storage means 52 in association with the user ID (user identification information).

[0123] Furthermore, when the electronic seal registration and management means 51 receives a user ID from the screen display means 21 of the work server 20, it also performs the process of sending the seal impression data stored in the seal impression storage means 52 associated with the received user ID to the screen display means 21 of the work server 20.

[0124] The seal impression storage means 52 stores the user's seal impression data in association with the user ID (user identification information).

[0125] <Detailed configuration of user terminal 60>

[0126] The drag operation determination means 61 repeatedly executes a process to determine whether or not an object (in this first embodiment, the stamp object) is being dragged. Detection of whether or not a drag operation is in progress is usually not possible with a single message, but rather with a combination of multiple messages. For example, if the input means 71 is a mouse, an event indicating that the left mouse button has been pressed is obtained. If the object (in this first embodiment, the stamp object) is at the position where the button was pressed, and an event indicating that the left mouse button has been released has not been obtained, i.e., the left mouse button is still being pressed, then a drag operation is in progress.

[0127] The authentication information acquisition means 62 performs a process to acquire authentication information used to determine whether or not to permit a drop operation that completes the intended action (in this first embodiment, the act of stamping an electronic seal) without requiring any operation other than the drag operation by the user, while the drag operation determination means 61 has determined that a drag operation is in progress and this determination is maintained. As shown in Figure 1, this authentication information acquisition means 62 includes the following means 62A to 62D.

[0128] The facial image acquisition means 62A performs a process to acquire facial image data of the user during a drag operation using the camera 72 as authentication information.

[0129] The coordinate acquisition means 62B performs a process to acquire the coordinate information of the current drag operation instruction position of the object being dragged (in this first embodiment, the seal impression object). The drag operation instruction position is the mouse cursor position, as already described in detail in the description of the screen display means 21.

[0130] The passage determination means 62C uses the coordinate information acquired by the coordinate acquisition means 62B to perform a process to determine whether the drag operation instruction position of the object (in this first embodiment, the seal object) has passed through a designated passage area, or a process to determine whether multiple designated passage areas have passed through in the correct order. The designated passage areas are, as already described in detail in the user information registration means 31, areas of designated spatial image data, or areas of designated passage image data or designated passage drawing targets. Information on these areas to be passed through is selected and specified by the user and stored in the passage information storage means 41C, so the screen display means 21 acquires it from the passage information storage means 41C and passes it to the passage determination means 62C.

[0131] When determining whether a specified passage area has been passed using hierarchical spatial image data, the passage determination means 62C uses the coordinate information acquired by the coordinate acquisition means 62B to determine whether the drag operation instruction position of an object (in this first embodiment, the seal impression object 102) has passed through any of the spatial image data areas among the displayed spatial image data group (each area placed and displayed in the passage area arrangement unit 104 in Figures 3 and 4). If it is determined that the object has passed through, the display (the entire display of the passage area arrangement unit 104 in Figures 3 and 4) is transitioned from each area of ​​the higher-level spatial image data group (the currently displayed spatial image data group) to each area of ​​the lower-level spatial image data group. During this transition, the process is executed to determine whether the drag operation instruction position of the object being dragged (in this case, the seal impression object 102) has passed through the area of ​​the specified spatial image data.

[0132] As shown in Figures 3 and 4, first, each region of the spatial image data group (spatial image data 1A, 1B, 1C, 1D, 1E, 1F, 1G, 1H, 1J) of the highest level (first level) is arranged and displayed in the passage area arrangement section 104. This arrangement and display is mainly performed by the screen display means 21 of the work server 20, but of course, the normal browser function on the user terminal 60 side is ensured by the processing means 60A. In the example of Figures 3 and 4, the passage area arrangement section 104 is composed of nine regions, but it is not limited to nine, and the number of regions that make up the passage area arrangement section is arbitrary. For example, it may be 4×4=16 regions, 3×4=12 regions, 2×4=8 regions, etc. Also, the overall shape of the passage area arrangement section does not necessarily have to be a square or rectangle, but is arbitrary. Furthermore, it is not necessary to place spatial image data regions in all of the placement positions of each region (nine regions in the examples of Figures 3 and 4) provided within the pass-through region placement unit 104. For example, the number of regions actually placed can be varied for each layer, such as placing 3 regions in the first layer, 8 regions in the second layer, and 5 regions in the third layer. The pass-through region placement unit 104 merely reserves space on the screen layout.

[0133] When a user drags an object (in this case, a seal object 102) to any spatial image data area within the pass-through area placement unit 104, and the object (more precisely, the drag operation instruction position) remains within that area, each area of ​​the lower-level spatial image data group for the spatial image data of the area where the object was located is placed and displayed in the pass-through area placement unit 104. This transition process of displaying from a higher level (the currently displayed level) to a lower level is mainly performed by the pass-through determination means 62C of the user terminal 60, but at the same time, processing by the pass-through determination assistance means 32F of the authentication server 30 via the work server 20 (the process of acquiring the lower-level spatial image data group from the spatial image storage means 42) is also performed.

[0134] The passage determination means 62C determines whether the object being dragged (more precisely, the drag operation instruction position) has remained within a certain spatial image data area. Whether or not it has remained is determined by information on whether it has stayed in the same area for a predetermined time (e.g., 1 second) or longer, or by information on whether it has stayed in the same area for a predetermined number of times the coordinate information (the number of times the coordinate information of the drag operation instruction position has been obtained in a loop) or longer. Therefore, instantaneous passage (passing for less than or equal to a predetermined time, or less than or equal to a predetermined number of times) is not determined to be a period of stay. In addition, when a user is dragging an object to remain within a certain spatial image data area, they usually perform an operation to keep the object still (not move it). However, in the determination by the passage determination means 62C, the user does not necessarily need to perform an operation to keep the object still (not move it). Even if the object (more precisely, the drag operation instruction position) is slightly moved within the same area, it will be determined that it has remained within that area.

[0135] Furthermore, passing through a particular passage area (in this case, the area of ​​spatial image data) refers to passing through it during a drag operation; therefore, the user does not perform a drop operation within that area. The drop operation is performed only in the target area (in the examples of Figures 3 and 4, the target areas 103 and 113), that is, in the area where the drop operation is to be performed in order to complete the target action (in this case, the act of stamping an electronic seal). Consequently, for example, if the input means 71 is a mouse, the user must keep the object (more precisely, the drag operation instruction position) within a particular passage area (in this case, the area of ​​spatial image data) by holding down the left mouse button.

[0136] Specifically, as shown in Figure 3, if the user leaves the seal impression object 102 in the area of ​​the upper right spatial image data 1C in the passage area placement unit 104, then the passage determination means 62C places and displays each area of ​​the lower-level (second-level) spatial image data group (spatial image data 2A, 2B, 2C, 2D, 2E, 2F, 2G, 2H) for that spatial image data 1C in the passage area placement unit 104. In this case, the user usually performs the operation of leaving the seal impression object 102 in place (not moving) in the area of ​​the upper right spatial image data 1C, so each area of ​​the lower-level spatial image data group is placed in a position other than the position of the spatial image data 1C where the dwelling operation was performed (upper right position). This is because, since users typically remain still in that area, if the spatial image data area of ​​a lower-level hierarchy is placed at the same location as the spatial image data area 1C where the dwell operation occurred (the upper right position out of the nine locations in the example in Figure 3), there is a high probability that the user will continue to point to the same location repeatedly. In other words, while the user is thinking about where to move the impression object 102 next, it is highly likely that the system will judge that the user has dwelled at the same location (the upper right position). Therefore, to avoid this, the spatial image data area of ​​a lower-level hierarchy is placed at a location other than the location of the area where the dwell operation occurred.

[0137] Next, if the user moves the seal impression object 102 to the area of ​​spatial image data 2F in the lower left and leaves the seal impression object 102 in that area, then the areas of the lower-level (third-level) spatial image data groups (spatial image data 3A, 3B, 3C, 3D, 3E, 3F, 3G, 3H) of spatial image data 2F are placed and displayed in the passage area placement unit 104 at positions other than the area of ​​spatial image data 2F by the passage determination means 62C.

[0138] Furthermore, if the user moves the seal impression object 102 to the area of ​​spatial image data 3F in the left center and leaves the seal impression object 102 in that area, then the areas of the lower-level (fourth-level) spatial image data groups (spatial image data 4A, 4B, 4C, 4D, 4E, 4F, 4G, 4H) of spatial image data 3F will be placed and displayed in the passage area arrangement unit 104 at positions other than the area of ​​spatial image data 3F, according to the passage determination means 62C.

[0139] Then, suppose the user finds a designated spatial image data that they have previously selected from the spatial image data set in the fourth layer, and that this is spatial image data 4B. At this point, the user moves the seal impression object 102 to the designated spatial image data 4B, keeps the seal impression object 102 in that area, then passes through the passage area placement section 104, moves the seal impression object 102 to the target area 103, and drops the seal impression object 102 there.

[0140] In this case, if the passage determination means 62C confirms that the seal impression object 102 is staying within the area of ​​the designated spatial image data, which is the spatial image data 4B, it may display an indication that the seal impression object has passed through the designated spatial image data (designated passage area) (such as "OK" or "Passage confirmed"). If the passage determination means 62C confirms that the seal impression object 102 has passed through the area of ​​the designated spatial image data, the determination means 32G of the authentication server 30 performs identity authentication using the passage determination result (there are other determination factors besides the passage determination result). If the identity authentication result is satisfactory, the drop operation to the target area 103 to complete the target action (in this case, the act of stamping the electronic seal) is permitted. In other words, the act of stamping the electronic seal is permitted.

[0141] On the other hand, if the user drops the seal impression object 102 without passing through the designated spatial image data (designated passage area), the drop operation will not be permitted, and the act of stamping will not be recognized. Also, if the user progresses through the hierarchical spatial image data, moving down through the layers, and ultimately reaches the last layer, but cannot find the designated spatial image data they selected, it means that the selection of the spatial image data area in the higher layers they passed through was inappropriate. In this case, if the user drops the seal impression object 102 in the target area 103 or any other arbitrary location, the drop operation will not be permitted, so they should restart the drag operation from the beginning.

[0142] It should be noted that the concept of hierarchical spatial image organization described above has already been utilized in the dictionary system described in Patent Document 5 mentioned above. Therefore, the novelty of the present invention lies in the use of hierarchical spatial images for user authentication, that is, in performing user authentication using the result of a pass-through determination of whether or not the user has passed through the area of ​​a designated spatial image data. Finding and passing through one designated pass-through image data from among many pass-through image data can be a very time-consuming operation. However, when using hierarchical spatial images, the upper and lower hierarchies are constructed based on the relationships (belonging relationships) of the contents of each spatial image data, thus avoiding the operational difficulty of taking a long time to find the image. For example, the first hierarchical level can be a collection of spatial image data included in "General Space," and can consist of three spatial image data: "Urban Space," "House Space," and "Natural Space." Here, when the user passes through the area of ​​the "Urban Space" spatial image data, the second hierarchical level displays the areas of multiple spatial image data, such as "Town Intersection Space," "Supermarket Space," and "Police Box Space," as a group of spatial image data related to (belonging to) "Urban Space." If the user has pre-selected a designated spatial image data, and that data is of an orange, then the user can find and pass through, for example, spatial image data of an "urban space" or a "greengrocer's space." Alternatively, the user can find and pass through spatial image data of a "house," a "dining room," a "tabletop space," or a "refrigerator space." Therefore, the selection path of spatial image data to reach the designated spatial image data does not need to be just one; there can be multiple paths, and the user only needs to be able to pass through the area of ​​the designated spatial image data in the end. It should be noted that the final image, such as an orange, is conceptually closer to an image of an object than a spatial image, but for the sake of explanation, since it is an object that exists in space, it is also treated as a spatial image in this application.

[0143] The examples in Figures 3 and 4 above illustrate the case where user authentication is performed using hierarchical spatial image data. However, when user authentication is performed using pass-through image data other than spatial image data, or pass-through drawing objects, the drag operation shown in the examples in Figures 5 to 8 below is performed. Note that even when passing through the area of ​​a designated pass-through image data or designated pass-through drawing object, the object (more precisely, the drag operation instruction position) must remain in that area. Therefore, instantaneous passage is not judged as dwelling, so even if an instantaneous passage occurs through an area of ​​pass-through image data other than the designated pass-through image data, or an area of ​​a pass-through drawing object other than the designated pass-through drawing object, that passage will not be judged as a passage due to an incorrect selection, and will not negatively affect user authentication. In other words, if a user wants to pass through the area of ​​"8," and instantaneously passes through the adjacent area of ​​"9," this does not mean that "9" has been selected. If the user then instantaneously passes through "9" and then remains in the area of ​​"8," it will be considered the operation intended by the user.

[0144] Figure 5 shows an example of the arrangement display of the passage area arrangement unit 120, which is composed of multiple (in this example, nine) passage image data areas G1 to G9. Of these, the user has pre-selected the first designated passage image data G9, the second designated passage image data G2, and the third designated passage image data G4. The user passes the object (in this case, the seal impression object) through in this designated order and then drops the object in the target area. In this example, the order is specified up to three, but the number of specified orders is arbitrary; it can be up to two, four, five, or more. Specifying more orders increases the effectiveness of user authentication, but also increases the effort required from the user.

[0145] FIG. 6 shows an example of the layout display of the passage area arrangement unit 130 composed of areas of a plurality (in this example, 10) of passage drawing objects (numbers 0 to 9). Among these, the first designated passage drawing object F8 (number 8), the second designated passage drawing object F6 (number 6), the third designated passage drawing object F1 (number 1), and the fourth designated passage drawing object F7 (number 7) were pre-selected and specified by the user. The user passes an object (here, a shaded object) in the order of this specified order and then drops the object in the target area. Note that the passage drawing object is not limited to numbers, and for example, alphabetic characters (A, B, C,...), Greek characters (α, β, γ,...), hiragana characters (あ, い, う,...), katakana characters (ア, イ, ウ,...), symbols (×, ÷, %, +,...), Chinese characters (桜, 猫, 電車,...), etc. may be used. In short, any information that can be drawn on the screen without using image data is acceptable.

[0146] FIG. 7 shows an example of the layout display of the passage area arrangement unit 140 in which each passage area is arranged in a column. This passage area arrangement unit 140 is composed of areas of a plurality (in this example, 10) of passage drawing objects (numbers 0 to 9). Among these, the first designated passage drawing object F6 (number 6), the second designated passage drawing object F8 (number 8), the third designated passage drawing object F4 (number 4), and the fourth designated passage drawing object F1 (number 1) were pre-selected and specified by the user. The user passes an object (here, a shaded object) in the order of this specified order and then drops the object in the target area.

[0147] Figure 8 shows an example of the arrangement display of the pass-through area arrangement unit 150, which is composed of multiple columns (four columns in this example) in which each pass-through area is arranged. Note that the vertical and horizontal orientation can be reversed to use multiple rows instead of multiple columns. In this pass-through area arrangement unit 150, there are multiple pass-through drawing targets (10 in this example) in each column, and the unit is composed of 10 × 4 columns = 40 pass-through drawing target areas in total. The first column contains pass-through drawing target areas F1-0, F1-1, F1-2, ..., F1-9, the second column contains pass-through drawing target areas F2-0, ..., F2-9, the third column contains pass-through drawing target areas F3-0, ..., F3-9, and the fourth column contains pass-through drawing target areas F4-0, ..., F4-9. Of these, the user pre-selected the following drawing targets for passing through: F1-4 (number 4), F2-2 (number 2), F3-4 (number 4), and F4-7 (number 7). The user then passes the objects (in this case, the seal objects) through in this specified order and drops the objects in the target area.

[0148] The voice acquisition means 62D uses the microphone 73 to perform a process to acquire voice data of the user during a drag operation. This voice data may be voice data from a phone call or conversation (provided that the other party's voice data is not acquired), or it may be voice data from when the user deliberately speaks for the purpose of user authentication. Even in the latter case, the user is not required to perform any user operations using their hands or fingers for user authentication. If facial image data for facial recognition cannot be acquired due to a malfunction of the camera 72 or the like, and voice data for voiceprint authentication is acquired as a supplement, the user may be informed of this and prompted to speak.

[0149] The drop permission determination means 63 receives the user authentication result transmitted from the determination means 32G of the authentication server 30 via the work server 20 and the network 1, and executes a process to determine whether or not to permit the drop operation to complete the intended action (in this first embodiment, the act of affixing an electronic seal) according to this user authentication result.

[0150] In this case, if the drop permission determination means 63 does not permit the drop operation, it will inform the user who performed the drag-and-drop operation that authentication was unsuccessful by displaying a screen message such as, "We could not verify your identity. The drop operation is invalid." On the other hand, if the drop operation is permitted, it will display the dropped seal object 102 at the location where it was dropped.

[0151] <Flowchart of the identity verification process using the electronic seal system 10: Figure 2>

[0152] In Figure 2, first, the user operates the user terminal 60 and inputs basic information (such as a password for conventional basic authentication) and registered personal information for personal authentication (the user's facial image data, voice data, and information for the designated passage area) (Step S1). The input information is transmitted to the authentication server 30 via network 1, one of the work servers 20, and network 2. The authentication server 30 receives basic information and registered personal information transmitted from the user terminal 60 via the user information registration means 31. The received basic information (password, etc.) is stored in the basic information storage means 41A of the user information storage means 41 in association with the user ID (user identification information). The facial image data or facial feature quantities from the received registered personal information are stored in the facial authentication information storage means 41B of the user information storage means 41 in association with the user ID. The voiceprint data obtained by voiceprint analysis of the audio data is stored in the voiceprint storage means 41D of the user information storage means 41 in association with the user ID. The information for the designated passage area is stored in the passage information storage means 41C in association with the user ID and registered (step S2).

[0153] Next, the user operates the user terminal 60 to input or create new seal impression data to be used (step S3). The input or newly created seal impression data is transmitted to the electronic seal server 50 via network 1, one of the work servers 20, and network 2. The electronic seal server 50 receives the seal impression data transmitted from the user terminal 60 using the electronic seal registration and management means 51, associates it with the user ID, and stores it in the seal impression storage means 52 for registration (step S4).

[0154] Subsequently, when a user performs a task requiring a seal impression using one of the work servers 20, they log in to that work server 20 by entering their user ID and password on the user terminal 60 (step S5). The entered user ID and password are transmitted to the authentication server 30 via network 1, work server 20, and network 2. The authentication server 30 performs basic authentication using the information stored in the basic information storage means 41A via the basic authentication means 32A, and transmits the authentication result to the user terminal 60 via network 2, work server 20, and network 1 (step S6). Note that if multi-factor authentication is not performed, basic authentication using the password at the time of login does not need to be performed.

[0155] Next, when the user terminal 60 requests the display of the work screen (step S7), the screen display means 21 of the work server 20 transmits data for displaying the work screen (step S8), and the work screen (work screens 100, 110, etc., as shown in Figures 3 and 4) is displayed on the screen of the user terminal 60 (step S7). This work screen displays the electronic document to be stamped, the stamp impression object 102, and the pass-through area arrangement section 104 which arranges multiple pass-through areas. When the work screen is displayed, the camera 72 and microphone 73 are activated.

[0156] Then, the user terminal 60 uses the drag operation determination means 61 to determine whether or not it is currently dragging the seal impression object 102 (step S9). If it is determined that it is not currently dragging, the same determination process is repeated.

[0157] On the other hand, if it is determined in step S9 that a drag operation is in progress, the facial image acquisition means 62A of the user terminal 60 acquires facial image data of the user performing the drag operation, and transmits the acquired facial image data to the authentication server 30 via network 1, the work server 20, and network 2 (step S10). In this case, if multiple facial image data are to be acquired for user authentication during a single drag operation, the data is acquired repeatedly each time the timing for processing in step S10 occurs through a loop process. Also, if user authentication is performed with only one facial image data, acquisition is not required from the second time onward.

[0158] Then, when the authentication server 30 receives face image data transmitted by the face image acquisition means 62A of the user terminal 60, the face feature extraction means 32B extracts face features from the received face image data (step S11).

[0159] Furthermore, the authentication server 30 calculates a face feature score by comparing the face features extracted by the face feature extraction means 32B (face features of the user who performed the tragg operation) with the face features of the user pre-registered in the face authentication information storage means 41B (or the face features of the user extracted by the face feature extraction means 32B from the user's face image data pre-registered in the face authentication information storage means 41B) using the face feature comparison means 32C (step S12). Note that the face features of the user pre-registered in the face authentication information storage means 41B (or the user's face image data pre-registered in the face authentication information storage means 41B) refer to the face features (or face image data) stored in the face authentication information storage means 41B in association with the user ID (user identification information) entered during login in step S5 as described above. In this case, when multiple facial image data are acquired and user authentication is performed during a single drag operation, multiple facial image data are sequentially transmitted by the facial image acquisition means 62A of the user terminal 60, the facial feature quantities are sequentially extracted by the facial feature quantity extraction means 32B, and the facial feature quantity comparison means 32C sequentially calculates facial feature quantity scores. The facial feature quantity comparison means 32C also performs a process to calculate an average facial feature quantity score by averaging all the facial feature quantity scores calculated up to that point. Therefore, the average facial feature quantity score is updated each time new facial image data is acquired and new facial feature quantity scores are calculated.

[0160] Next, the user terminal 60 acquires the user's voice data during the drag operation using the voice acquisition means 62D, and transmits the acquired voice data to the authentication server 30 via network 1, the work server 20, and network 2 (step S13). At this time, if voice data acquisition has not yet started, it is started, and if voice data acquisition has already started, it is continued.

[0161] Then, when the authentication server 30 receives the voice data transmitted by the voice acquisition means 62D of the user terminal 60, the voiceprint analysis means 32D performs voiceprint analysis using the received voice data and generates voiceprint data of the user during the drag operation (step S14).

[0162] Next, the authentication server 30 calculates a voiceprint score by comparing the voiceprint data generated by the voiceprint analysis by the voiceprint analysis means 32D (voiceprint data of the user who performed the tragg operation) with the voiceprint data of the person who performed the operation that is pre-registered in the voiceprint storage means 41D (step S15). The voiceprint data of the person who performed the operation that is pre-registered in the voiceprint storage means 41D refers to the voiceprint data stored in the voiceprint storage means 41D in association with the user ID (user identification information) entered during login in step S5 as described above.

[0163] Then, the user terminal 60 acquires coordinate information of the drag operation instruction position (such as the mouse cursor position during the drag operation) using the coordinate acquisition means 62B (step S16).

[0164] Furthermore, the user terminal 60 uses the pass-through determination means 62C to perform a pass-through determination process to determine whether or not it has passed through the designated pass-through area, using the coordinate information acquired by the coordinate acquisition means 62B. The pass-through determination result is then transmitted to the authentication server 30 via network 1, the work server 20, and network 2 (step S17), and the authentication server 30 receives the pass-through determination result using the pass-through determination auxiliary means 32F (step S18). Specifically, it determines whether or not it has passed through the area of ​​the designated spatial image data, or whether or not it has passed through the area of ​​the designated pass-through image data or the designated pass-through drawing target. However, the details of this pass-through determination process have already been described in detail in the description of the pass-through determination means 62C, so a detailed explanation is omitted here. The information on the designated pass-through area has already been transmitted to the user terminal 60 by the screen display means 21 of the work server 20.

[0165] Furthermore, when determining whether or not a user has passed through a specified spatial image data area, layered spatial image data is used, so data is communicated between the user terminal 60's pass-through determination means 62C and the authentication server 30's pass-through determination assistance means 32F (steps S17, S18). Specifically, the pass-through determination means 62C requests the transmission of a lower-level spatial image data group (including identification information of the lower level to be transmitted) (step S17), and the pass-through determination assistance means 32F obtains the lower-level spatial image data group related to the transmission request from the spatial image storage means 42 and transmits it to the pass-through determination means 62C (step S18).

[0166] Next, the user terminal 60 uses the drag operation determination means 61 to determine whether or not it is currently dragging the seal impression object 102 (step S19). If it is determined that it is currently dragging, the process returns to step S10, and steps S10 to S19 are repeated until it is determined that it is not currently dragging. Therefore, this loop process is executed while a drag operation is in progress.

[0167] On the other hand, if it is determined in step S19 that a drag operation is not in progress (i.e., a drop operation has been performed), the drop permission determination means 63 of the user terminal 60 sends a request to the authentication server 30 via network 1, the work server 20, and network 2 to send a request for the transmission of the user authentication result (step S20). The authentication server 30 uses the determination means 32G to perform user authentication processing using information obtained from at least one of the facial feature comparison means 32C, the voiceprint comparison means 32E, and the pass-through determination assistance means 32F, and transmits the user authentication result to the user terminal 60 via network 2, the work server 20, and network 1 (step S21). The user terminal 60 then uses the drop permission determination means 63 to receive the user authentication result transmitted by the determination means 32G of the authentication server 30, and makes a decision on whether to permit the drop operation (i.e., whether to permit the electronic seal impression) according to the received user authentication result, and displays a message to that effect on the screen if it is not permitted (step S20).

[0168] <Flowchart of the process for determining whether a specified spatial image data region has been passed: Figure 9>

[0169] In Figure 9, when the user terminal 60 receives display data for the work screen transmitted from the screen display means 21 of the work server 20, the work screen (work screens 100, 110, etc., as shown in Figures 3 and 4) is displayed on the screen (step S7 in Figure 2 as described above). This work screen is provided with a pass-through area arrangement unit 104 that arranges and displays multiple pass-through areas (each area of ​​the spatial image data group of the highest level).

[0170] Next, the user terminal 60 acquires coordinate information of the current drag operation instruction position (such as the mouse cursor position during the drag operation) using the coordinate acquisition means 62B (step S16 in Figure 2 as described above).

[0171] Then, the user terminal 60 performs the following pass-through determination process using the pass-through determination means 62C (step S17 in Figure 2 mentioned above).

[0172] Specifically, first, it is determined whether or not the specified spatial image data region has already been passed by the time of the previous determination (step S1701). The information on whether or not the specified spatial image data region has been passed is stored in the memory (main memory is sufficient) of the user terminal 60. If it is determined that the region has already been passed, the process proceeds to the next determination in the loop processing steps S10 to S19 in Figure 2 described above. On the other hand, if it is determined that the region has not yet been passed, it is determined whether or not any of the spatial image data regions have been passed (step S1702). If it is determined that none of the spatial image data regions have been passed, the process proceeds to the next determination in the loop processing.

[0173] On the other hand, if it is determined in step S1702 that a region of any spatial image data has been passed through, it is determined whether or not that passage is through the region of the specified spatial image data (step S1703). If it is determined that the passage is through the region of the specified spatial image data, the passage determination result is sent to the authentication server 30 (step S1704). Then, the process proceeds to the next determination in the loop processing.

[0174] On the other hand, if step S1703 determines that the specified spatial image data area has not been passed through, it is determined whether or not there is a group of spatial image data at a lower (immediately below) level for the spatial image data of the passed area (step S1705). If it is determined that there is a group of spatial image data at a lower (immediately below) level, a request is made to the authentication server 30 to send the group of spatial image data at the lower level, and each area of ​​the group of spatial image data at the lower level obtained from the authentication server 30 is placed and displayed in the area placement unit 104 for passing through (step S1706). Then, the process proceeds to the next determination in the loop processing.

[0175] In addition, information regarding whether or not there is a group of spatial image data at a lower (immediately below) level for a given spatial image data has already been sent to the user terminal 60 along with the spatial image data in question. However, if this information has not been sent, the user may still request the authentication server 30 to send the group of spatial image data at a lower (immediately below) level, even if the existence of such a group remains unknown.

[0176] On the other hand, if in step S1705 it is determined that there are no spatial image data sets in the lower (immediately below) hierarchy, even if the user is indeed the legitimate user, the user's selection is considered incorrect. In this case, a request is made to the authentication server 30 to send the spatial image data set in the highest hierarchy, and each region of the spatial image data set in the highest hierarchy obtained from the authentication server 30 is placed and displayed in the pass-through area placement unit 104 (step S1707). Then, the process proceeds to the next decision in the loop processing. Consequently, the user restarts the drag operation of the seal object 102 from the beginning.

[0177] Furthermore, if the presence or absence of spatial image data for a lower (immediately below) hierarchy remains unknown, and a request for transmission of spatial image data for a lower (immediately below) hierarchy is made to the authentication server 30, and the authentication server 30's pass-through determination assisting means 32F determines that there is no spatial image data for a lower (immediately below) hierarchy, the pass-through determination assisting means 32F should transmit the spatial image data for the highest hierarchy to the pass-through determination means 62C of the user terminal 60.

[0178] <Flowchart of the process for determining passage through specified passage image data other than specified spatial image data, or the area to be drawn for specified passage: Figure 10>

[0179] In Figure 10, when the user terminal 60 receives display data for the work screen transmitted from the screen display means 21 of the work server 20, the work screen is displayed on the screen (step S7 in Figure 2 as described above). This work screen is provided with a pass-through area arrangement section (for example, pass-through area arrangement sections 120, 130, 140, 150, etc. in Figures 5 to 8) that arranges and displays multiple pass-through areas (including designated pass-through areas). Multiple pass-through areas are each area of ​​multiple pass-through image data or each area of ​​multiple pass-through drawing targets.

[0180] Next, the user terminal 60 acquires coordinate information of the current drag operation instruction position (such as the mouse cursor position during the drag operation) using the coordinate acquisition means 62B (step S16 in Figure 2 as described above).

[0181] Then, the user terminal 60 performs the following pass-through determination process using the pass-through determination means 62C (step S17 in Figure 2 mentioned above).

[0182] In other words, first, it is determined whether the last designated passage area (designated passage image data or designated passage drawing target area) has already been passed by the time of the previous determination (step S1711). If it is determined that it has already been passed, the process proceeds to the next determination in the loop processing steps S10 to S19 in Figure 2 described above.

[0183] On the other hand, if it is determined in step S1711 that the user has not yet passed through a designated area, information is obtained about the designated areas that have been passed through (designated area image data or area to be drawn for designated area passage), that is, information that the user has passed through up to the Kth designated area (designated area image data or area to be drawn for designated area passage) (step S1712). This information that the user has passed through up to the Kth area is stored in the memory of the user terminal 60 (main memory is sufficient). If the user has not passed through the first area, K=0.

[0184] Next, it is determined whether or not the system has passed through any of the passage areas (passage image data or areas to be drawn for passage) (step S1713). If it is determined that the system has not passed through, the process proceeds to the next determination in the loop. On the other hand, if it is determined that the system has passed through, it is determined whether or not the passage is through the (K+1)th designated passage area (specified passage image data or area to be drawn for passage) (step S1714).

[0185] Then, in step S1714, if it is determined that the user has not passed through the (K+1)th designated passage area, it means that the user has passed through a passage area that is not one of the designated passage areas that should have been passed through, so the information of the passed area is cleared and K=0 (step S1715). Then, the process proceeds to the next decision in the loop. Consequently, the user restarts the drag operation of the seal object 102 from the beginning.

[0186] On the other hand, if step S1714 determines that the (K+1)th designated passage area has been passed, it is determined whether the (K+1)th designated passage area is the last designated passage area in the sequence (step S1716). If it is determined that it is the last designated passage area, a passage determination result indicating that all designated passage areas have been passed in the specified order is sent to the authentication server 30 (step S1717). Then, the process proceeds to the next determination in the loop.

[0187] On the other hand, if step S1716 determines that the user has not passed through the last designated pass area, the information that the user has passed through the (K+1)th designated pass area is saved in the user terminal 60's memory (main memory is acceptable) (step S1718). Then, the process proceeds to the next decision in the loop.

[0188] <Effects of the First Embodiment>

[0189] According to this first embodiment, the following effects are obtained. That is, since the electronic seal system 10 is equipped with authentication information acquisition means 62, when a user performs a drag operation to carry out a target action (in this first embodiment, the act of stamping an electronic seal), authentication information can be acquired during that drag operation. Therefore, authentication can be performed using the authentication information acquired during this drag operation, and based on the result of that authentication, it can be determined whether or not to permit a drop operation to complete the target action (in this case, the act of stamping).

[0190] In this case, the user is not required to perform any operations other than dragging to obtain authentication information. Furthermore, since the user is performing the dragging operation in order to perform the intended action (in this case, the act of stamping), the user is not being required to perform the dragging operation solely for the purpose of obtaining authentication information. In other words, the user logs into the system in order to complete the intended action, and the dragging operation is an essential operation for performing that action. Therefore, the authentication information can be obtained at the same time as the user is performing such an essential operation. This reduces the effort required of the user.

[0191] Furthermore, the authentication information acquisition means 62 acquires authentication information used to determine whether or not to permit a drop operation to complete the intended action (in this case, the act of stamping) while a drag operation is being performed to carry out the intended action. Therefore, if no drag operation is performed, authentication information is not acquired, and authentication information is acquired only when a drag operation is performed. For this reason, unlike the user authentication process performed when logging into the system, that is, the user authentication process performed uniformly regardless of what action the user takes, the user authentication process can be performed only for the necessary actions. Consequently, it is possible to avoid performing the same user authentication process for simple actions that are not so important as for important actions, thus reducing the burden on the user.

[0192] Furthermore, even if user authentication is performed when logging into the system, if the user leaves their desk and goes away from the computer for any reason afterward, it is impossible to prevent someone else from operating the computer and committing fraudulent acts because the user's authentication has already been completed. In contrast, the electronic seal system 10 obtains authentication information and performs user authentication during the drag operation to carry out the intended action (in this case, the act of affixing an electronic seal), thus preventing fraudulent acts due to a change in the operator after login.

[0193] Furthermore, the electronic seal system 10 includes a facial recognition information storage means 41B, a facial image acquisition means 62A, a facial feature extraction means 32B, and a facial feature comparison means 32C. Therefore, it can acquire the user's facial image data during a drag operation, extract facial features from that image data, and calculate a facial feature score for the user during the drag operation by comparing it with pre-registered facial features or facial features extracted from pre-registered facial image data. This facial feature score can then be used by the determination means 32G to perform identity verification. In this case, the user is only performing a drag operation to carry out the intended action (in this case, the act of stamping), and the acquisition of facial image data for identity verification is performed without the user being particularly aware of it, thus reducing the effort required of the user.

[0194] Furthermore, the facial feature comparison means 32C can calculate an average facial feature score using multiple facial feature scores obtained from multiple facial image data of the user during a drag operation, thereby enabling stable user authentication.

[0195] Furthermore, since the electronic seal system 10 is equipped with a voiceprint storage means 41D, a voice acquisition means 62D, a voiceprint analysis means 32D, and a voiceprint comparison means 32E, it can acquire the user's voice data during a drag operation, generate voiceprint data from that voice data, calculate a voiceprint score for the user during the drag operation by comparing it with pre-registered voiceprint data, and use this voiceprint score to perform authentication by the determination means 32G. In this case, the user is only performing a drag operation to carry out the intended action (in this case, the act of stamping), and is not forced to perform any special operations to acquire the voice data used for authentication, thus reducing the effort required from the user.

[0196] Furthermore, the electronic seal system 10 is equipped with a passage information storage means 41C, a passage determination means 62C, and a passage determination auxiliary means 32F. Therefore, during a drag operation, it can obtain a passage determination result indicating whether or not the user has passed through a designated passage area determined based on the user's drag path, and use this passage determination result to perform user authentication using the determination means 32G. In this case, the user is only performing a drag operation to carry out the intended action (in this case, the act of stamping), and is not forced to perform any special operations (operations other than the drag operation) for user authentication, thus reducing the effort required of the user.

[0197] Furthermore, the above pass-through determination result can be determined by using hierarchical spatial image data to determine whether or not the user has passed through the specified spatial image data area. In other words, the user can navigate through the display of each area of ​​the spatial image data group from higher to lower levels to reach the specified spatial image data area. In this case, since the spatial image data in the upper and lower levels are related to each other (there is a content-based inclusion relationship), the user can reach the specified spatial image data area by dragging while considering this content-based relationship. Therefore, although there is only one specified spatial image data to pass through, the user's selection is made by dragging among many hierarchical spatial image data areas, thus increasing the authentication effect. In other words, as already explained in detail, if there is only one specified pass-through area, the order of passage cannot be used as a determination factor, so the authentication effect cannot be increased unless the number of pass-through areas (number of options) displayed on the screen is increased. However, if the number of pass-through areas is too large, it will take the user time to find the specified pass-through area, and it will become difficult to display all pass-through areas on one screen. However, these problems can be solved by using hierarchical spatial image data.

[0198] Furthermore, when using a physical seal, the user manually moves the seal to the designated area on the paper. However, in the electronic seal system 10, this manual movement is replaced by dragging the seal object 102 on the screen. Consequently, the user will perceive the act of applying the electronic seal via dragging as very natural, and this natural dragging operation can be used to achieve user authentication.

[0199] [Second Embodiment]

[0200] Figure 11 shows the overall configuration of the problem system 200, which is an information processing system according to the second embodiment of the present invention, and Figure 12 shows a flowchart illustrating the user authentication process by the problem system 200. Figure 13 is an explanatory diagram of the answering process on the problem screen 280 in the problem system 200.

[0201] <Configuration of Problem System 200>

[0202] The problem system 200 of this second embodiment contains many components in common with the electronic seal system 10 of the first embodiment, and many parts perform the same functions. Therefore, the same reference numerals are used for the same parts, and detailed explanations are omitted. The following explanation will focus on the differences.

[0203] In the electronic seal system 10 of the first embodiment described above, work servers 20 (work servers 20A, 20B, 20C, ... for the first, second, third, ...) were provided as shown in Figure 1. However, in the problem system 200 of this second embodiment, instead of those work servers 20, problem servers 220 (problem servers 220A, 220B, 220C, ... for the first, second, third, ...) are provided as shown in Figure 11.

[0204] Furthermore, while the electronic seal system 10 of the first embodiment was provided with an electronic seal server 50 as shown in Figure 1, the problem system 200 of this second embodiment is not provided with an electronic seal server 50.

[0205] In Figure 11, each question server 220 presents various questions, such as language learning or language proficiency test questions, to users operating the user terminal 60, receives answers, and performs scoring. Each question server 220 is composed of one or more computers. That is, one question server 220 may be composed of one computer, or it may be composed of multiple computers, such as a web server, web application server, or database server.

[0206] Each question server 220 is equipped with processing means 221 that performs various processes related to question generation, answer reception, and scoring, and this processing means 221 includes a screen display means 222 and a scoring means 223. Each question server 220 is also equipped with question storage means 224 that stores question data and correct answer data.

[0207] Each of the means 222 and 223 included in the processing means 221 is implemented by a central processing unit (CPU) located inside the problem server 220, and one or more programs that define the operating procedures of this CPU. Furthermore, the problem storage means 224 can be, for example, a hard disk drive (HDD), a solid-state drive (SSD), or the like.

[0208] The screen display means 222 receives a request to display a problem screen transmitted from the user terminal 60 via the network 1, and uses the problem data stored in the problem storage means 224 to create display data for various problem screens, such as the problem screen 280 in Figure 13, and transmits it to the user terminal 60 via the network 1. As a result, the various problem screens are displayed on the display means 70 of the user terminal 60.

[0209] Since the first problem server 220A, the second problem server 220B, the third problem server 220C, ... handle problems with different content, the screen display means 222 of each problem server 220 executes the process of displaying a problem screen with different content.

[0210] In Figure 13, the problem screen 280 is the screen for displaying a rearrangement problem for learning Chinese. This problem screen 280 displays multiple (four in this example) draggable answer objects 281, 282, 283, and 284. The user rearranges these answer objects 281, 282, 283, and 284 by dragging them and drops them into the target area 285 to submit their answer.

[0211] In the example shown in Figure 13, not all of the answer objects 281, 282, 283, and 284 are considered answer objects in this invention; only one answer object (in this example, the answer object 282 labeled "Teacher") is considered an answer object in this invention. That is, while the answer object 282 is being dragged, the authentication information acquisition means 62 acquires various authentication information.

[0212] More specifically, during the drag operation of the solution object 282, the face image acquisition means 62A acquires the user's face image data.

[0213] Furthermore, when the user performs a drag operation on the answer object 282, they utter the phrase (in this example, "teacher") while performing the drag operation. During this drag operation, the voice acquisition means 62D acquires the user's voice data.

[0214] Furthermore, when dragging the answer object 282, the user passes it through the pass-through area placement unit 286, which contains the areas of multiple pass-through drawing targets. In the example in Figure 13, the designated pass-through drawing targets are the areas with the numbers "2", "6", "4", and "8" drawn on them, and the user passes through each area in this order. It would also be possible to have all four answer objects 281, 282, 283, and 284 pass through the pass-through area placement unit 286, but this would be time-consuming for the user. Additionally, it is sufficient to confirm the user's selection using only one answer object 282. Moreover, since the answer is not valid unless all four answer objects 281, 282, 283, and 284 are dragged, it is ultimately sufficient to perform user authentication using only one answer object 282. Therefore, in this example, authentication information is acquired only during the drag operation of one answer object 282.

[0215] Furthermore, if the authentication process by the authentication server 30's determination means 32G fails to confirm the user's identity, the user will not be allowed to drop the answer object 282, and will not be permitted to answer the rearrangement problem.

[0216] The scoring means 223 receives the answer data dropped into the target area 285 on the screen of the user terminal 60 via the network 1, scores it using the correct answer data stored in the problem storage means 224, and transmits the scoring result to the user terminal 60 via the network 1.

[0217] Furthermore, the scoring means 223 also evaluates the user's pronunciation of words using the user's voice data acquired by the voice acquisition means 62D, and transmits the evaluation results to the user terminal 60 via the network 1.

[0218] <Problem System 200's User Authentication Process Flow: Figure 12>

[0219] The processing in steps S201 and S202 in Figure 12 is the same as the processing in steps S1 and S2 in Figure 2 of the first embodiment, the only difference being that it goes through the problem server 220, whereas in the first embodiment it goes through the work server 20.

[0220] Since the problem system 200 does not handle electronic seals, Figure 12 does not include the processes S3 and S4 in steps S4 of Figure 2 of the first embodiment.

[0221] The processes in steps S203 and S204 in Figure 12 are the same as the processes in steps S5 and S6 in Figure 2 of the first embodiment, differing only in that they go through the problem server 220, whereas in the first embodiment they go through the work server 20.

[0222] The processes in steps S205 and S206 in Figure 12 correspond to the processes in steps S7 and S8 in Figure 2 of the first embodiment. In Figure 12, the screen display means 222 displays the problem screen 280 (see Figure 13) and the problem and answer objects 282, whereas in Figure 2 of the first embodiment, the screen display means 21 displays the electronic documents to be stamped and the stamp impression objects 102 on the work screens 100 and 110 (see Figures 3 and 4).

[0223] The processing in steps S207 to S219 in Figure 12 is the same as the processing in steps S9 to S21 in Figure 2 of the first embodiment, the only difference being that it goes through the problem server 220, whereas in the first embodiment it goes through the work server 20. In Figure 12, step S218 determines whether to allow or disallow the act of answering a problem by drop operation, whereas in Figure 2 of the first embodiment, step S20 determines whether to allow or disallow the act of affixing an electronic seal by drop operation. However, both are the same in that they determine whether or not to allow the drop operation.

[0224] Figure 12 shows the processes in steps S220 and S221, which are not performed in Figure 2 of the first embodiment. Specifically, the processing means 60A of the user terminal 60 transmits the user's answer data to the problem server 220 via the network 1 (step S220). The problem server 220 uses the scoring means 223 to score the answer data received from the user terminal 60 using the correct answer data stored in the problem storage means 224, and transmits the scoring result to the user terminal 60 via the network 1 (step S221). The user terminal 60 receives the scoring result and displays it on the screen (step S220).

[0225] <Effects of the second embodiment>

[0226] According to this second embodiment, in addition to obtaining the same effects as the first embodiment, when the user answers a question, the user's voice data can be acquired in a natural way by having the user pronounce words related to the question as part of the answer, and this can be used for user authentication.

[0227] [Third Embodiment]

[0228] Figure 14 shows the overall configuration of the asset transfer system 300, which is an information processing system according to the third embodiment of the present invention, and Figure 15 shows a flowchart illustrating the flow of the user authentication process by the asset transfer system 300. Figure 16 is an explanatory diagram of a transfer operation, which is an execution act involving asset transfer on the transfer screen 380 of the asset transfer system 300, and Figure 17 is an explanatory diagram of a product purchase operation, which is an execution act involving asset transfer on the product purchase screen 390 of the asset transfer system 300.

[0229] <Configuration of Asset Transfer System 300>

[0230] The asset transfer system 300 of this third embodiment includes many components in common with the electronic seal system 10 of the first embodiment, and many parts perform the same functions. Therefore, the same reference numerals are used for the same parts, and detailed explanations are omitted. The following explanation will focus on the differences.

[0231] In the electronic seal system 10 of the first embodiment described above, as shown in Figure 1, there were work servers 20 (first, second, third, ... work servers 20A, 20B, 20C, ...). However, in the asset transfer system 300 of this third embodiment, instead of those work servers 20, there are multiple asset transfer servers 320, as shown in Figure 14. In the example in Figure 14, there is a first asset transfer server, which is a transfer server 320A, a second asset transfer server, which is a remittance server 320B, a third asset transfer server, which is a product purchase server 320C, and so on.

[0232] Furthermore, while the electronic seal system 10 of the first embodiment was provided with an electronic seal server 50 as shown in Figure 1, the asset transfer system 300 of this third embodiment is not provided with an electronic seal server 50.

[0233] In Figure 14, each asset transfer server 320 (320A, 320B, 320C, ...) executes processing to receive execution actions involving asset transfer from users operating the user terminal 60, and each is composed of one or more computers. That is, one asset transfer server 320 may be composed of one computer, or it may be composed of multiple computers such as a web server, web application server, database server, etc.

[0234] Here, execution actions involving asset transfers are actions that result in the transfer of the user's assets (including exchange-related actions such as exchanging yen for dollars), such as transfers to financial institution accounts, transfers between accounts, buying and selling of securities or other financial products, e-commerce, and actions involving in-game charges.

[0235] Each asset transfer server 320 is equipped with processing means 321 that performs various processes related to execution actions involving asset transfer, and this processing means 321 includes a screen display means 322 and an asset transfer means 323. The screen display means 322 displays the execution object and displays the arrangement of the passage area (including the designated passage area). The asset transfer means 323 performs the process of actually moving the asset when the user's execution action is approved. Each of the means 322 and 323 included in the processing means 321 is implemented by a central processing unit (CPU) located inside the asset transfer server 320 and one or more programs that define the operating procedures of this CPU.

[0236] Specifically, for example, in the case of the first asset transfer server, the transfer server 320A, the processing means 321 performs various processes related to the transfer operation, the screen display means 322 displays the execution object for executing the transfer operation and displays the arrangement of the transit area (including the designated transit area), and the asset transfer means 323 is a remittance means that transfers funds from the account of the financial institution that holds the user's assets to the account of the financial institution to which the funds are transferred. The transfer server 320A also includes an account information storage means 324 that stores data related to the user's assets (deposit and withdrawal data, balance data, etc.) and a registered account storage means 325 that stores information on registered accounts such as past transfer destinations. For example, the account information storage means 324 and the registered account storage means 325 can be hard disk drives (HDDs), solid state drives (SSDs), etc.

[0237] In Figure 16, the transfer screen 380 is an asset transfer screen displayed by the screen display means 322 of the transfer server 320A. This transfer screen 380 displays an execution object 381, a target area 382 which shows registered accounts (such as accounts with a history of past transfers) that can be selected as candidates when specifying the account of the financial institution to which the transfer will be made by dragging the execution object 381, and a passage area arrangement unit 383 which arranges each area of ​​the spatial image data group that the execution object 381 will pass through when dragging it to the target area 382. In the example shown in Figure 16, the user enters and specifies the transfer amount = 12,800 yen on another screen, then on the transfer screen 380, drags the execution object 381, which contains the user's account information (W Bank α account, account holder: Kukan Taro) and the transfer amount = 12,800 yen, passes it through the area of ​​the specified spatial image data in the passage area placement unit 383, and then drops it in the target area 382, ​​which shows the account of the recipient financial institution (Y Bank γ account, Kukan Hanako).

[0238] In Figure 17, the product purchase screen 390 is an asset transfer screen displayed by the screen display means 322 of the product purchase server 320C, which is a third asset transfer server. The product purchase screen 390 displays a plurality of execution objects 391 corresponding to each product, a target area 392 that shows the cart when the user selects and drags an execution object 391 corresponding to the product they wish to purchase from among these execution objects 391 and adds it to the cart to indicate their intention to purchase the product, and a passage area arrangement section 393 that arranges each area of ​​the spatial image data group through which the execution object 391 is passed when dragging it to the target area 392.

[0239] In the example shown in Figure 17, it is not necessary to pass the pass-through area placement unit 393 through the execution object 391 corresponding to all products. The pass-through area placement unit 393 is only passed through when dragging the execution object 391 corresponding to a high-priced product (in this example, the Kukando copier, 128,000 yen). Therefore, for inexpensive products (for example, products under 100,000 yen), personal authentication using the pass-through determination result is not performed. As a result, while the execution object 391 is being dragged, various authentication information is acquired by the authentication information acquisition means 62. However, the pass-through determination result by the pass-through determination means 62C is only obtained for high-priced products, and the pass-through determination process by the pass-through determination means 62C is not performed for inexpensive products.

[0240] More specifically, during the drag operation of execution objects 381 and 391 (see Figures 16 and 17), the face image acquisition means 62A acquires the user's face image data.

[0241] Furthermore, during the drag operation of the execution objects 381 and 391, the voice acquisition means 62D acquires the user's voice data.

[0242] Furthermore, when dragging the execution objects 381 and 391, the user passes them through the passage area placement units 383 and 393, which contain the respective regions of the spatial image data. At this time, the passage determination means 62C performs a passage determination process to determine whether or not the user has passed through the region of the specified spatial image data. Note that instead of using a passage determination process that utilizes hierarchical spatial image data, the user may also perform a passage determination process that determines whether or not the user has passed through the regions of several specified passage image data (excluding spatial image data) or the regions of several specified passage drawing targets (among several passage drawing targets) in the correct order.

[0243] Furthermore, if the authentication process by the authentication server 30's determination means 32G fails to confirm that the person is indeed the person in question, the drop operation of the execution objects 381 and 391 will not be permitted, and execution actions involving asset transfer, such as bank transfers and product purchases, will not be allowed.

[0244] <Flowchart of the identity verification process using the asset transfer system 300: Figure 15>

[0245] The processes in steps S301 and S302 in Figure 15 are the same as the processes in steps S1 and S2 in Figure 2 of the first embodiment, differing only in that they go through the asset transfer server 320, whereas in the first embodiment they go through the work server 20.

[0246] Since the asset transfer system 300 does not handle electronic seals, Figure 15 does not include the processes S3 and S4 in steps S4 of Figure 2 of the first embodiment.

[0247] The processes in steps S303 and S304 in Figure 15 are the same as the processes in steps S5 and S6 in Figure 2 of the first embodiment, differing only in that they go through the asset transfer server 320, whereas in the first embodiment they go through the work server 20.

[0248] The processes of steps S305 and S306 in FIG. 15 correspond to the processes of steps S7 and S8 in FIG. 2 of the first embodiment. In FIG. 15, the execution objects 381, 391, etc. are displayed on the asset transfer screens such as the transfer screen 380 (see FIG. 16) and the product purchase screen 390 (see FIG. 17) by the screen display means 322, while in FIG. 2 of the first embodiment, the screen display means 21 displays the electronic documents to be stamped and the stamping object 102 on the work screens 100, 110 (see FIGS. 3 and 4). This is the difference.

[0249] The processes of steps S307 to S319 in FIG. 15 are the same as the processes of steps S9 to S21 in FIG. 2 of the first embodiment, and the only difference is that they are passed through the asset transfer server 320, while in the first embodiment, they are passed through the work server 20. In FIG. 15, in step S318, a determination is made on whether to permit an execution act involving asset transfer by a drop operation, while in FIG. 2 of the first embodiment, in step S20, a determination is made on whether to permit the stamping act of the electronic seal by a drop operation. However, both are the same in that they are determinations on whether to permit the drop operation.

[0250] In FIG. 15, there are the processes of steps S320 and S321, which are not performed in FIG. 2 of the first embodiment. That is, the processing means 60A of the user terminal 60 transmits asset transfer data such as transfer data and product purchase data to the asset transfer server 320 via the network 1 (step S = 320). The asset transfer server 320 uses the asset transfer data received from the user terminal 60 by the fund transfer means 323 such as the remittance means to execute asset transfer processes such as remittance and deposit and withdrawal (step S321).

[0251] <Effect of the Third Embodiment>

[0252] According to such a third embodiment, the same effects as those of the first embodiment can be obtained. In addition, the execution act involving asset transfer is usually a more nerve-consuming and important act compared to the reference of the balance and the confirmation of product specifications that are performed before it. Therefore, for such an act, individual authentication can be performed.

[0253] [Form of Variation]

[0254] Note that the present invention is not limited to the above-described embodiments, and modifications and the like within the scope that can achieve the object of the present invention are included in the present invention.

[0255] For example, the electronic seal system 10, the problem system 200, and the asset transfer system 300 of the above-described embodiments are configured to connect a user terminal 60 and various servers via a network 1, but the information processing system of the present invention may have a stand-alone configuration. [Industrial Applicability]

[0256] As described above, the information processing system and program of the present invention are suitable for use in, for example, an electronic seal system, a problem system for language learning and language testing, or an asset transfer system that executes processes such as transfers to accounts at financial institutions, transfers between accounts, trading of securities or other financial products, e-commerce transactions, actions involving charging in games, or other processes involving asset transfers. [Description of Signs]

[0257] 10 Electronic seal system which is an information processing system 20 Screen display means <![CDATA[ ]]> 31 User information registration means 32 Authentication means 32B Facial feature extraction means <00![CDATA[ ]]> 32C Facial feature comparison means 32D Voiceprint analysis means 32E Voiceprint comparison means 32G Determination means 41 User information storage means 41B Facial authentication information storage means 41C Passing information storage means 41D Voiceprint storage means 42 Spatial image storage means 43 Passing image storage means 52 Stamp impression storage means 61. Means for determining when drag operation is in progress 62. Means for obtaining authentication information 62A Face image acquisition method 62B Coordinate acquisition means 62C Passage judgment means 62D Audio Acquisition Method 72 Cameras 73 Microphones 102 Seal impression object 200 Information Processing Systems: Problem Systems 222 Screen display means 224 Problem memory means 282 Solution Objects 300 Asset transfer system, which is an information processing system. 322 Screen display means 381,391 Execution Objects

Claims

1. An information processing system comprising a computer that performs processing to accept the user's intended action through on-screen operations, A user information storage means that stores registered personal information for identity verification, pre-registered by the user, in association with user identification information, A screen display means that performs a process to display on the screen objects used for drag-and-drop operations to perform the aforementioned purpose action by the user, A drag operation determination means that performs a process to determine whether or not the object is being dragged, The drag operation determination means determines that a drag operation is in progress, and while this determination is maintained, the authentication information acquisition means executes a process to acquire authentication information used to determine whether or not to permit a drop operation that completes the aforementioned objective action without requiring any operation other than the drag operation by the user. The system includes an authentication means that uses the authentication information obtained by the authentication information acquisition means and the registered personal information stored in the user information storage means in association with the user identification information used by the user who performed the drag operation when logging in to perform an authentication process to determine whether the user who performed the drag operation is the person corresponding to the user identification information used when logging in. The user information storage means is The aforementioned registered personal information includes a facial recognition information storage means that stores the personal's facial image data or facial feature quantities extracted from this facial image data in association with user identification information. The authentication information acquisition means is: The authentication information includes a face image acquisition means that performs a process to acquire face image data of the user during a drag operation using a camera. The aforementioned authentication means is A face feature extraction means that performs a process to extract the face feature quantities of the user during a drag operation from the face image data acquired by the face image acquisition means, or in addition to this process, performs a process to extract the face feature quantities of the user from the face image data of the user if the user's face image data is stored in the face authentication information storage means as registered user information. A face feature comparison means performs a process to compare the face features of the user during the drag operation extracted by the face feature extraction means with the face features of the user stored as registered personal information in the face recognition information storage means or the face features of the user extracted by the face feature extraction means from the user's face image data stored as registered personal information in the face recognition information storage means, and calculates a face feature score indicating the degree of match. This system includes a determination means that, based on the level of the facial feature score calculated by this facial feature comparison means, determines whether the user who performed the drag operation is the person corresponding to the user identification information used during login. The system includes a seal impression storage means that stores the seal impression data of the electronic seal used by the user, associated with user identification information. The aforementioned screen display means is The system is configured to perform a drag-and-drop operation to allow the user to apply an electronic seal, by placing a seal impression object on the screen and then displaying a target area on the screen where the user can drop the seal impression object to complete the application. The user information storage means is In addition to the aforementioned facial recognition information storage means, When the aforementioned seal impression object is dragged, the user verifies their identity by having the object pass through a designated pass-through area selected in advance by the user from among multiple pass-through areas displayed on the screen. The registered personal information includes a pass-through information storage means that stores, in association with user identification information, the designated pass-through image data or designated pass-through drawing object selected from among multiple pass-through image data or multiple pass-through drawing objects, or the pass-through order when passing through multiple designated pass-through areas in order. The aforementioned screen display means is The system is configured to also perform the process of arranging and displaying multiple of the aforementioned passage areas on the screen. The authentication information acquisition means is: In addition to the aforementioned facial image acquisition means, A coordinate acquisition means that performs a process to acquire coordinate information of the current drag operation instruction position of the dragged seal object, This system includes a passage determination means that uses the coordinate information acquired by the coordinate acquisition means to perform a process to determine whether the drag operation instruction position of the seal impression object has passed through the designated passage area, or a process to determine whether a plurality of the designated passage areas have passed through in order. The determination means of the authentication means is The system is configured to use the results of the facial feature score determination calculated by the facial feature comparison means and the pass determination results by the pass determination means to determine whether the user who performed the drag operation is the person corresponding to the user identification information used during login. An information processing system characterized by the following:

2. The aforementioned information storage means for passing through is, The system is configured to store identification information for the specified spatial image data used for passage, which is selected in advance by the user from among a group of spatial image data of multiple hierarchical levels in which the spatial image data of the upper and lower levels are related to each other. The aforementioned screen display means is The system is configured to also perform the process of arranging and displaying on the screen the regions formed by each of the spatial image data sets of the highest hierarchical level, which serve as multiple transit regions. The means for determining whether to pass the authentication information acquisition means is: The system is configured to use the coordinate information acquired by the coordinate acquisition means to determine whether the drag operation instruction position of the seal impression object has passed through any of the regions of the spatial image data among the regions of the spatial image data group currently being displayed. If it is determined that the object has passed through a region, the display is transitioned from the regions of the higher-level spatial image data group to the regions of the lower-level spatial image data group. During this transition, the system also performs a process to determine whether the drag operation instruction position of the seal impression object being dragged has passed through the region of the specified spatial image data. The information processing system according to feature 1.

3. The aforementioned facial image acquisition means is The configuration is such that the process of acquiring the aforementioned facial image data is repeatedly executed during a single drag operation. The facial feature extraction means of the authentication means is The configuration includes a process to extract the facial features of the user during the drag operation from each of the multiple facial image data acquired by the facial image acquisition means, or, in addition to this process, a process to extract the facial features of the user from the user's facial image data if the user's facial image data is stored in the facial recognition information storage means as registered user information. The facial feature comparison means of the authentication means is The system is configured to perform a process in which each of the multiple facial features of the user during a drag operation, extracted by the facial feature extraction means, is compared with the user's facial features stored as registered user information in the facial recognition information storage means, or with the user's facial features extracted by the facial feature extraction means from the user's facial image data stored as registered user information in the facial recognition information storage means, calculates multiple facial feature scores indicating the degree of match, and calculates an average facial feature score by averaging these multiple facial feature scores. The determination means of the authentication means is The system is configured to use the results of the determination of whether the average facial feature score calculated by the facial feature comparison means is high or low, and the results of the pass judgment means, to determine whether the user who performed the drag operation is the person corresponding to the user identification information used during login. The information processing system according to claim 1 or 2, characterized in that it is the same as described in claim 1 or 2.

4. An information processing system comprising a computer that performs processing to accept the user's intended action through on-screen operations, A user information storage means that stores registered personal information for identity verification, pre-registered by the user, in association with user identification information, A screen display means that performs a process to display on the screen objects used for drag-and-drop operations to perform the aforementioned purpose action by the user, A drag operation determination means that performs a process to determine whether or not the object is being dragged, The drag operation determination means determines that a drag operation is in progress, and while this determination is maintained, the authentication information acquisition means executes a process to acquire authentication information used to determine whether or not to permit a drop operation that completes the aforementioned objective action without requiring any operation other than the drag operation by the user. The system includes an authentication means that uses the authentication information obtained by the authentication information acquisition means and the registered personal information stored in the user information storage means in association with the user identification information used by the user who performed the drag operation when logging in to perform an authentication process to determine whether the user who performed the drag operation is the person corresponding to the user identification information used when logging in. The user information storage means is When dragging the aforementioned object, identity verification is performed by passing it through a designated pass-through area that has been pre-selected by the user from among multiple pass-through areas displayed on the screen. The registered identity information includes a pass-through information storage means that stores, in association with user identification information, the designated pass-through image data or designated pass-through drawing object selected from among multiple pass-through image data or multiple pass-through drawing objects, or the pass-through order when passing through multiple designated pass-through areas in order. The aforementioned screen display means is The system is configured to also perform the process of arranging and displaying multiple of the aforementioned passage areas on the screen. The authentication information acquisition means is: A coordinate acquisition means that performs a process to acquire coordinate information of the current drag operation instruction position of the object being dragged, This system includes a passage determination means that uses the coordinate information acquired by the coordinate acquisition means to perform a process to determine whether the drag operation instruction position of the dragged object has passed through the designated passage area, or a process to determine whether a plurality of the designated passage areas have passed through in order. The aforementioned authentication means is The system is configured to use the pass-through determination result from the pass-through determination means to perform a process to determine whether the user who performed the drag operation is the person corresponding to the user identification information used during login. An information processing system characterized by the following:

5. The system includes a seal impression storage means that stores the seal impression data of the electronic seal used by the user, associated with user identification information. The aforementioned screen display means is The system is configured to display a seal impression object on the screen as an object used for drag-and-drop operations to perform the act of applying an electronic seal by the user, and to display a target area on the screen where the user can drop this seal impression object to complete the application. The information processing system according to feature 4.

6. The aforementioned screen display means is The system is configured to perform drag-and-drop operations to execute actions that result in the transfer of funds to a financial institution account, transfers between accounts, buying and selling of securities or other financial products, e-commerce, in-game charges, or other asset transfers. These actions involve placing execution objects on the screen that represent cash, virtual currency, financial products, or other financial assets, the source account or owner of the financial assets, the object of an e-commerce transaction, a game character, or other representations related to asset transfers. The system also includes a process to display a target area on the screen for dropping these execution objects to complete the execution action. The information processing system according to feature 4.

7. An information processing system comprising a computer that performs processing to accept the user's intended action through on-screen operations, A user information storage means that stores registered personal information for identity verification, pre-registered by the user, in association with user identification information, A screen display means that performs a process to display on the screen objects used for drag-and-drop operations to perform the aforementioned purpose action by the user, A drag operation determination means that performs a process to determine whether or not the object is being dragged, The drag operation determination means determines that a drag operation is in progress, and while this determination is maintained, the authentication information acquisition means executes a process to acquire authentication information used to determine whether or not to permit a drop operation that completes the aforementioned objective action without requiring any operation other than the drag operation by the user. The system includes an authentication means that uses the authentication information obtained by the authentication information acquisition means and the registered personal information stored in the user information storage means in association with the user identification information used by the user who performed the drag operation when logging in to perform an authentication process to determine whether the user who performed the drag operation is the person corresponding to the user identification information used when logging in. The user information storage means is The aforementioned registered personal information includes a voiceprint storage means that stores the personal voiceprint data in association with user identification information. The authentication information acquisition means is: It is configured to include a voice acquisition means that performs a process to acquire voice data of the user during a drag operation using a microphone, The aforementioned authentication means is A voiceprint analysis means performs voiceprint analysis using the voice data of the user during a drag operation acquired by the voice acquisition means and executes a process to create voiceprint data for voiceprint authentication. A voiceprint comparison means performs a process to compare the voiceprint data of the user during the drag operation obtained by the voiceprint analysis means with the voiceprint data of the user stored in the voiceprint storage means as registered user information, and calculates a voiceprint score indicating the degree of match. This system includes a determination means that performs a process to determine whether the user who performed the drag operation is the person corresponding to the user identification information used during login, based on the level of the voiceprint score calculated by this voiceprint comparison means. An information processing system characterized by the following:

8. The system includes a seal impression storage means that stores the seal impression data of the electronic seal used by the user, associated with user identification information. The aforementioned screen display means is The system is configured to perform a drag-and-drop operation to allow users to apply an electronic seal, by placing a seal impression object on the screen and then displaying a target area on the screen where the user can drop this seal impression object to complete the application. The information processing system according to feature 7.

9. The user information storage means is In addition to the voiceprint memory means, The aforementioned registered personal information includes a facial recognition information storage means that stores the personal's facial image data or facial feature quantities extracted from this facial image data in association with user identification information. The authentication information acquisition means is: In addition to the aforementioned voice acquisition means, The authentication information includes a face image acquisition means that performs a process to acquire face image data of the user during a drag operation using a camera. The aforementioned authentication means is In addition to the voiceprint analysis means and the voiceprint comparison means, A face feature extraction means that performs a process to extract the face feature quantities of the user during a drag operation from the face image data acquired by the face image acquisition means, or in addition to this process, performs a process to extract the face feature quantities of the user from the face image data of the user if the user's face image data is stored in the face authentication information storage means as registered user information. This system includes a face feature comparison means that performs a process to compare the face features of the user during the drag operation extracted by the face feature extraction means with the face features of the user stored as registered user information in the face recognition information storage means, or the face features of the user extracted by the face feature extraction means from the user's face image data stored as registered user information in the face recognition information storage means, and calculate a face feature score indicating the degree of match. The determination means of the authentication means is The system is configured to perform a process to determine whether the user who performed the drag operation is the person corresponding to the user identification information used during login, based on the level of the voiceprint score calculated by the voiceprint comparison means and the level of the facial feature score calculated by the facial feature comparison means, or based on the level of authenticity calculated by integrating the voiceprint score and the facial feature score. The information processing system according to feature 8.

10. The user information storage means is In addition to the facial recognition information storage means and the voiceprint storage means, When the aforementioned seal impression object is dragged, the user verifies their identity by having the object pass through a designated pass-through area selected in advance by the user from among multiple pass-through areas displayed on the screen. The registered personal information includes a pass-through information storage means that stores, in association with user identification information, the designated pass-through image data or designated pass-through drawing object selected from among multiple pass-through image data or multiple pass-through drawing objects, or the pass-through order when passing through multiple designated pass-through areas in order. The aforementioned screen display means is The system is configured to also perform the process of arranging and displaying multiple of the aforementioned passage areas on the screen. The authentication information acquisition means is: In addition to the facial image acquisition means and the voice acquisition means, A coordinate acquisition means that performs a process to acquire coordinate information of the current drag operation instruction position of the dragged seal object, This system includes a passage determination means that uses the coordinate information acquired by the coordinate acquisition means to perform a process to determine whether the drag operation instruction position of the seal impression object has passed through the designated passage area, or a process to determine whether a plurality of the designated passage areas have passed through in order. The determination means of the authentication means is The determination of whether the facial feature score is high or low calculated by the facial feature comparison means, the determination of whether the voiceprint score is high or low calculated by the voiceprint comparison means, and the pass-through determination result by the pass-through determination means are used, Alternatively, using the result of determining the high or low value of the combined score of the voiceprint score and the facial feature score, and the pass-through determination result by the pass-through determination means, The system is configured to perform a process to determine whether the user who performed the drag operation is the person corresponding to the user identification information used during login. The information processing system according to feature 9.

11. It includes a question storage means for storing question data for language learning or language proficiency tests to be presented to the user, The aforementioned screen display means is The system is configured to use the problem data stored in the problem storage means to place answer objects representing words or other answer elements on the screen as objects used for drag-and-drop operations for the user to perform an answer action involving vocalization to word rearrangement problems, fill-in-the-blank problems, or other language-related problems, and to display a target area on the screen for dropping these objects to complete the answer action. The voice acquisition means of the authentication information acquisition means is The system is configured to use a microphone to capture audio data when a user speaks while performing a drag operation and provides an answer. The information processing system according to feature 7.

12. The user information storage means is In addition to the voiceprint memory means, The aforementioned registered personal information includes a facial recognition information storage means that stores the personal's facial image data or facial feature quantities extracted from this facial image data in association with user identification information. The authentication information acquisition means is: In addition to the aforementioned voice acquisition means, The authentication information includes a face image acquisition means that performs a process to acquire face image data of the user during a drag operation using a camera. The aforementioned authentication means is In addition to the voiceprint analysis means and the voiceprint comparison means, A face feature extraction means that performs a process to extract the face feature quantities of the user during a drag operation from the face image data acquired by the face image acquisition means, or in addition to this process, performs a process to extract the face feature quantities of the user from the face image data of the user if the user's face image data is stored in the face authentication information storage means as registered user information. This system includes a face feature comparison means that performs a process to compare the face features of the user during the drag operation extracted by the face feature extraction means with the face features of the user stored as registered user information in the face recognition information storage means, or the face features of the user extracted by the face feature extraction means from the user's face image data stored as registered user information in the face recognition information storage means, and calculate a face feature score indicating the degree of match. The determination means of the authentication means is The system is configured to perform a process to determine whether the user who performed the drag operation is the person corresponding to the user identification information used during login, based on the level of the voiceprint score calculated by the voiceprint comparison means and the level of the facial feature score calculated by the facial feature comparison means, or based on the level of authenticity calculated by integrating the voiceprint score and the facial feature score. The information processing system according to feature 11.

13. A program for causing a computer to function as an information processing system according to any one of claims 1 to 12.

Images