Information processing device, information processing method, and recording medium

Abstract

An information processing device according to the present disclosure comprises an acquisition unit, a determination unit, and an error process execution unit. The acquisition unit acquires at least one among a peripheral image captured by a terminal device during the execution of a predetermined process and a peripheral sound collected by the terminal device. The determination unit determines whether the terminal device is at a designated position on the basis of at least one among the peripheral image and the peripheral sound. The error process execution unit performs an error process when it is determined that the terminal device is not at the designated position.

Description

Information processing device, information processing method, and recording medium

[0001] The present disclosure relates to an information processing device, an information processing method, and a recording medium.

[0002] A technology related to the present disclosure is disclosed in Patent Document 1. Patent Document 1 discloses a technology for confirming that a prospective user is near a predetermined location. In this technology, a presence confirmation application device reads a passcode projected at a predetermined location using a projection device and transmits the read passcode to a presence confirmation device. The presence confirmation device compares the received passcode with a pre-registered passcode, and if they match, determines that the presence confirmation application device is within a certain range of the predetermined location.

[0003] Japanese Patent Application Laid-Open No. 2022-181755

[0004] In recent years, various types of online processing have become possible. In such online processing, a user may be required to operate a terminal device at a designated location to perform the processing. Therefore, there is a demand for a technique to confirm whether the user complies with the request.

[0005] The technology disclosed in Patent Literature 1 can be used to confirm that a terminal device is at a specified location. However, with the technology disclosed in Patent Literature 1, it is not possible to detect fraudulent activity in which, after it has been confirmed that the terminal device is at the specified location, the user moves to another location with the terminal device and performs processing at another location.

[0006] In view of the above-mentioned problems, an example of an objective of the present disclosure is to provide a technology that suppresses the problem of a user performing a predetermined process online at a location other than the designated location.

[0007] The information processing device of the present disclosure has an acquisition means for acquiring at least one of a peripheral image captured by a terminal device executing a predetermined process and a peripheral sound collected by the terminal device, a determination means for determining whether the terminal device is at a designated position based on at least one of the peripheral image and the peripheral sound, and an error processing execution means for performing error processing when it is determined that the terminal device is not at the designated position.

[0008] In addition, the information processing method disclosed herein includes one or more computers acquiring at least one of a surrounding image captured by a terminal device executing a specified process and a surrounding sound collected by the terminal device, determining whether the terminal device is at a specified position based on at least one of the surrounding image and the surrounding sound, and performing error processing if it is determined that the terminal device is not at the specified position.

[0009] Furthermore, the recording medium of the present disclosure records a program that causes a computer to function as: an acquisition means that acquires at least one of a peripheral image captured by a terminal device executing a predetermined process and a peripheral sound collected by the terminal device; a determination means that determines whether the terminal device is at a specified position based on at least one of the peripheral image and the peripheral sound; and an error processing execution means that performs error processing when it is determined that the terminal device is not at the specified position.

[0010] According to one aspect of the present disclosure, a technique is realized that prevents a problem in which a user performs a predetermined online process at a location other than a designated location.

[0011] FIG. 1 is a diagram showing an example of a functional block diagram of an information processing device according to the present disclosure. FIG. 2 is a flowchart showing an example of a processing flow of an information processing device according to the present disclosure. FIG. 3 is a diagram showing an example of a hardware configuration of an information processing device according to the present disclosure. FIG. 4 is a diagram showing an example of a functional block diagram of a system including an information processing device according to the present disclosure. FIG. 5 is a diagram showing an example of information processed by an information processing device according to the present disclosure. FIG. 6 is a diagram showing another example of information output by a terminal device according to the present disclosure. FIG. 7 is a diagram showing another example of information output by a terminal device according to the present disclosure. FIG. 8 is a flowchart showing another example of a processing flow of an information processing device according to the present disclosure. FIG. 9 is a diagram showing another example of a functional block diagram of a system including an information processing device according to the present disclosure. FIG. 10 is a diagram showing another example of a functional block diagram of a system including an information processing device according to the present disclosure. FIG. 11 is a diagram showing another example of information processed by an information processing device according to the present disclosure.

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In this disclosure, the drawings relate to one or more embodiments. In all drawings, similar components are designated by similar reference numerals, and descriptions thereof will be omitted as appropriate.

[0013] <First embodiment> Fig. 1 is a functional block diagram showing an overview of an information processing device 10. As shown in Fig. 1, the information processing device 10 has an acquisition unit 11, a determination unit 12, and an error processing execution unit 13. These functional units execute the processing of the flowchart in Fig. 2.

[0014] In S10, the acquisition unit 11 acquires at least one of a surrounding image captured by the terminal device while the terminal device is executing a predetermined process and a surrounding sound collected by the terminal device while the terminal device is executing the predetermined process. The predetermined process is, for example, a process that is requested to be executed at a specified position. In S11, the determination unit 12 determines whether the terminal device is at the specified position based on at least one of the surrounding image and the surrounding sound. In S12, the error processing execution unit 13 performs error processing if it determines that the terminal device is not at the specified position.

[0015] In this way, the information processing device 10 has a feature of acquiring at least one of a peripheral image captured by a "terminal device executing a predetermined process" and a peripheral sound collected by a "terminal device executing a predetermined process." In other words, the information acquired by the information processing device 10 is not necessarily any peripheral image captured by a terminal device or any peripheral sound collected by a terminal device. The information processing device 10 targets to acquire at least one of a peripheral image captured by a "terminal device executing a predetermined process" and a peripheral sound collected by a "terminal device executing a predetermined process."

[0016] The information processing device 10 then determines whether the terminal device is at the specified position based on at least one of the acquired peripheral image and peripheral sound. When the information processing device 10 acquires the content as described above, the determination result indicates whether the "terminal device executing a predetermined process" is at the specified position.

[0017] Then, when it is determined that the "terminal device currently executing a predetermined process" is not at the specified location, the information processing device 10 executes an error process.

[0018] According to such an information processing device 10, it is possible to prevent the problem of a user performing a predetermined online process at a location other than the designated location.

[0019] Second Embodiment Overview An information processing device 10 according to a second embodiment is a specific implementation of the configuration of the control device 10 according to the first embodiment. The following will describe it in detail.

[0020] "Hardware Configuration" First, an example of the hardware configuration of the information processing device 10 will be described. Each functional unit of the information processing device 10 is realized by any combination of hardware and software. Those skilled in the art will understand that there are various variations in the realization method and device. Software includes programs that are pre-stored in the device before shipping, and programs downloaded from recording media such as CDs (Compact Discs) or servers on the Internet.

[0021] FIG. 3 is a block diagram illustrating an example of the hardware configuration of an information processing device 10. As shown in FIG. 3, the information processing device 10 has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The information processing device 10 does not necessarily have to have the peripheral circuit 4A. Note that the information processing device 10 may be composed of multiple devices that are physically and / or logically separated. In this case, each of the multiple devices may have the above hardware configuration.

[0022] The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to mutually transmit and receive data. The processor 1A is, for example, a processing unit such as a CPU or a graphics processing unit (GPU). The memory 2A is, for example, a random access memory (RAM) or a read-only memory (ROM). The input / output interface 3A includes interfaces for acquiring information from input devices, external devices, external servers, external sensors, cameras, etc., and interfaces for outputting information to output devices, external devices, external servers, etc. The input / output interface 3A also includes an interface for connecting to a communication network such as the Internet. Examples of input devices include a keyboard, mouse, microphone, physical buttons, touch panel, etc. Examples of output devices include a display, speaker, printer, mailer, etc. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

[0023] "Functional Configuration" Next, the functional configuration of the information processing device 10 will be described in detail. Fig. 1 shows an example of a functional block diagram of the information processing device 10. As shown in the figure, the information processing device 10 has an acquisition unit 11, a determination unit 12, and an error processing execution unit 13.

[0024] 4, the information processing device 10 is a server 1. The information processing device 10 is connected to a terminal device 2 via a communication network 3 such as the Internet.

[0025] The terminal device 2 is a terminal operated by a user. The terminal device 2 may be a portable terminal that can be carried around. The terminal device 2 has a camera function, a microphone function, a communication function, an input / output function, a calculation function, etc. The terminal device 2 may also have other functions. The terminal device 2 is, for example, a smartphone, a tablet terminal, a personal computer, a smart watch, a mobile phone, a game console, etc., but is not limited to these. The hardware configuration of such a terminal device 2 can be realized using any widely known technology.

[0026] Returning to FIG. 1, the acquisition unit 11 acquires at least one of a surrounding image captured by the terminal device 2 while the terminal device 2 is executing a predetermined process and a surrounding sound collected by the terminal device 2 while the terminal device 2 is executing a predetermined process.

[0027] The "predetermined process" is a process that is requested to be executed at a specified location. All "predetermined processes" that appear below refer to processes that are requested to be executed at a specified location. The predetermined process may be a process that is executed via a specified application installed on the terminal device 2. Examples of the predetermined process include, but are not limited to, "online calls," "processes that accept information input via a UI (user interface) screen and send the information to a server," and "processes that output content to an output device and allow a user to view it."

[0028] The "online call" may be a video call in which images and audio are transmitted and received, or a call in which only audio is transmitted and received. In recent years, online calls have been used to conduct online interviews, online meetings, online lessons, online exams, and various other events. In some cases, a user is required to operate the terminal device 2 at a designated location to participate in such an event. The predetermined application installed on the terminal device 2 may be an application dedicated to predetermined processing for participating in an online interview, online meeting, online lesson, online exam, or various other events using online calls.

[0029] The "process of accepting information input via a UI screen and transmitting the information to a server" is used, for example, when inputting necessary information and transmitting it to a server to perform various procedures (such as membership registration, contract cancellation, and various other applications). The user may be required to operate the terminal device 2 at a designated location and perform such procedures at the designated location. Furthermore, as part of a campaign, recreation, or the like, the user may be given a mission that can be completed by inputting predetermined information via the UI screen at a designated location. The predetermined application installed on the terminal device 2 may be an application dedicated to a predetermined process for performing various procedures online. Furthermore, the predetermined application installed on the terminal device 2 may be an application dedicated to a predetermined process for participating in the campaign, recreation, or the like described above.

[0030] The "process of outputting content to an output device and allowing a user to view it" is a process of outputting video, still images, or audio to an output device for viewing by the user. The user may be required to operate the terminal device 2 at a specified location and view the content at the specified location. For example, the user may be given a mission to view a specified video at a specified location as part of a campaign, recreation, or the like. The specified application installed on the terminal device 2 may be an application dedicated to a specified process for viewing specified content online. Furthermore, the specified application installed on the terminal device 2 may be an application dedicated to a specified process for participating in the campaign, recreation, or the like described above.

[0031] The "designated location" is a location that has been designated in advance as a location where the above-mentioned predetermined process is to be executed. At least one location is designated in advance as a designated location and registered in the information processing device 10. Which location is designated as the designated location depends on the content of the predetermined process and is not particularly limited. For example, a public place such as a ward office or city hall may be designated as the designated location. Furthermore, a location where a mission is to be executed in a campaign, recreation, or the like may be designated as the designated location.

[0032] The "surrounding image" is an image of the surroundings of the terminal device 2 that is executing a predetermined process. It may also be an image of the surroundings generated by the terminal device 2 that is executing a predetermined process. The surrounding image may be composed of multiple images in chronological order.

[0033] The "surrounding sound" is sound collected by the terminal device 2 while the terminal device 2 is executing a predetermined process and generated around the terminal device 2 while the terminal device 2 is executing a predetermined process.

[0034] While executing the predetermined process, the terminal device 2 executes at least one of a process of generating a peripheral image by turning on the camera function and a process of collecting peripheral sound by turning on the microphone function. Then, the terminal device 2 transmits at least one of the peripheral image and the peripheral sound to the information processing device 10. Note that if the terminal device 2 has multiple cameras (in-camera and out-camera), it may generate a peripheral image by turning on only one of the cameras, or may generate a peripheral image by turning on all of the cameras.

[0035] The terminal device 2 can continue the predetermined process while it is executing the process. For example, the terminal device 2 may constantly capture moving images to generate a peripheral image while it is executing the predetermined process. The terminal device 2 may also constantly collect peripheral sounds using a microphone while it is executing the predetermined process. The terminal device 2 may also capture still images at predetermined time intervals to generate a peripheral image while it is executing the predetermined process. The terminal device 2 may also capture moving images of a predetermined length at predetermined time intervals to generate a peripheral image while it is executing the predetermined process. The terminal device 2 may also collect peripheral sounds of a predetermined length at predetermined time intervals while it is executing the predetermined process.

[0036] The terminal device 2 transmits at least one of a peripheral image and a peripheral sound to the information processing device 10 in real time. Then, the acquisition unit 11 acquires at least one of the peripheral image and the peripheral sound transmitted from the terminal device 2. The terminal device 2 and the information processing device 10 are connected to each other so that they can communicate with each other.

[0037] In addition, an application dedicated to the above-mentioned specified processing installed on the terminal device 2 may cause the terminal device 2 to perform the above-mentioned processing of generating the peripheral image, collecting peripheral sound, and transmitting at least one of the peripheral image and the peripheral sound to the information processing device 10.

[0038] In addition, another application different from the application dedicated to the predetermined process may be installed on the terminal device 2. The other application may then cause the terminal device 2 to perform the above-described processes of generating a peripheral image, collecting peripheral sound, and transmitting at least one of the peripheral image and the peripheral sound to the information processing device 10. In this case, the other application executes the process in cooperation with the application dedicated to the predetermined process. Specifically, the other application is configured to obtain information indicating that the predetermined process is being executed from the application dedicated to the predetermined process. The other application is then configured to recognize, based on the information, that the application dedicated to the predetermined process is executing the predetermined process. The other application then causes the terminal device 2 to execute the above-described processes of generating a peripheral image, collecting peripheral sound, and transmitting at least one of the peripheral image and the peripheral sound to the information processing device 10 while executing the predetermined process.

[0039] "Acquisition" includes at least one of the following: a device going to retrieve data or information stored in another device or storage medium (active acquisition), and a device inputting data or information output from another device (passive acquisition). Examples of active acquisition include making a request to another device and receiving a response, and accessing and reading information from another device or storage medium. An example of passive acquisition is receiving information that is distributed (or transmitted, push notification, etc.). Furthermore, "acquisition" may also mean selecting and acquiring data or information from received data or information, or selecting and receiving distributed data or information.

[0040] The determination unit 12 determines whether the terminal device 2 is located at a specified position based on at least one of the surrounding image and the surrounding sound.

[0041] The determination unit 12 determines whether the terminal device 2 is at the designated position based on the result of a detection process that includes at least one of a process of detecting a target object from a peripheral image and a process of detecting a target sound from peripheral audio. For example, when the determination unit 12 detects a target object from a peripheral image, the determination unit 12 can determine that the terminal device 2 is at the designated position. Furthermore, when the determination unit 12 detects a target sound from peripheral audio, the determination unit 12 can determine that the terminal device 2 is at the designated position.

[0042] The determination unit 12 can execute any one of the following determination examples 1 to 3.

[0043] Determination Example 1 In this example, a person or object present at a specified position is registered in advance in the information processing device 10 as a target object.

[0044] An "object" is a person or an object that can be a landmark of a specified location. At least one of a person and an object can be an object. One person or one object may be registered as an object corresponding to one specified location. Alternatively, multiple people or multiple objects may be registered as objects corresponding to one specified location.

[0045] For example, a person who is supposed to be at a specified location or a person who has a high tendency to be at a specified location may be registered as an object. Specifically, a person who works at a specified location or a person who performs work at a specified location may be registered as an object.

[0046] In addition, objects artificially installed at a specified position may be registered as targets. Specifically, facilities, machines, tools, etc. installed at a specified position may be registered as targets. In addition, paper media such as posters, paintings, other artworks, etc. installed at a specified position may be registered as targets. Paper media may display artificial information that can be distinguished from others by appearance, such as a one-dimensional code or a two-dimensional code. In addition, other objects artificially installed at a specified position may be registered as targets.

[0047] Additionally, objects that exist naturally at a specified position may be registered as targets. Specifically, trees, mountains, rivers, plants, etc. may be registered as targets.

[0048] The feature amounts of the appearance of the object are registered in advance in the information processing device 10. The determination unit 12 detects the object from the peripheral image based on the feature amounts. Then, when the determination unit 12 detects the object from the peripheral image, it determines that the terminal device 2 is at the specified position.

[0049] Note that a pass condition for determining that "the terminal device 2 is at the designated position" may be set in advance. Then, the determination unit 12 may determine that the terminal device 2 is at the designated position when the pass condition is satisfied.

[0050] An example of a passing condition is "person A has been detected." Another example of a passing condition is "both person A and person B have been detected." Another example of a passing condition is "both person A and object C have been detected." Another example of a passing condition is "at least one of person A and object C has been detected." In this way, the passing condition is defined by one object or a combination of multiple objects.

[0051] The above-mentioned "detection of both a first object and a second object" is a concept that includes at least one of the following. As described above, the peripheral image is composed of a plurality of images in chronological order. - The first object and the second object are detected simultaneously in one image - The second object is detected in another image within a predetermined time after the first object is detected in one image - The first object is detected in another image within a predetermined time after the second object is detected in one image

[0052] A person or an object may be registered in advance as a target object, and the determination unit 12 may determine that the terminal device 2 is at the designated position when the target object is detected in the surrounding image.

[0053] Alternatively, at least one of a person and an object may be registered in advance as a plurality of objects. Then, the determination unit 12 may determine that the terminal device 2 is at the specified position when a predetermined object or a combination of predetermined objects is detected in the peripheral image.

[0054] Determination Example 2 In this example, a sound present at a specified position is registered in advance in the information processing device 10 as a target sound.

[0055] A "target sound" is a sound that can be a landmark of a specified position. At least one of a human voice and other sounds can be the target sound. A single sound may be registered as the target sound. Alternatively, multiple sounds may be registered as targets.

[0056] For example, the voiceprint of a person who is supposed to be at the designated location or who is likely to be at the designated location may be registered as the target sound. Specifically, the voiceprint of a person who works at the designated location or who performs work at the designated location may be registered as the target sound.

[0057] Alternatively, sounds that are artificially played at a designated position may be registered as target sounds. For example, music or announcements played at a designated position at a fixed time may be registered as target sounds. Alternatively, music or announcements that are constantly played at a designated position may be registered as target sounds. Alternatively, sounds emitted by objects artificially installed at a designated position may be registered as target sounds. Alternatively, sounds of frequencies that humans cannot hear may be output at a designated position. These sounds may then be registered as target sounds.

[0058] Alternatively, a sound that is naturally occurring at a specified position may be registered as the target sound. For example, a sound emitted by an object that is naturally present at the specified position may be registered as the target sound.

[0059] The feature amount of the target sound is registered in advance in the information processing device 10. The determination unit 12 detects the target sound from the surrounding sounds based on the feature amount. Then, when the determination unit 12 detects the target sound from the surrounding sounds, it determines that the terminal device 2 is at the specified position.

[0060] Note that a pass condition for determining that "the terminal device 2 is at the designated position" may be set in advance. Then, the determination unit 12 may determine that the terminal device 2 is at the designated position when the pass condition is satisfied.

[0061] An example of a passing condition is "the voice of person A has been detected." Another example of a passing condition is "the voices of both person A and person B have been detected." Another example of a passing condition is "the voice of both person A and music D have been detected." Another example of a passing condition is "the voice of person A and at least one of music D has been detected." In this way, the passing condition is defined by one sound or a combination of multiple sounds.

[0062] The above phrase "both the first target sound and the second target sound have been detected" means "the second target sound has been detected within a predetermined time after the first target sound has been detected."

[0063] One sound may be registered in advance as one target sound, and the determination unit 12 may determine that the terminal device 2 is at the designated position when the target sound is detected from the surrounding sounds.

[0064] Alternatively, multiple target sounds may be registered in advance. Then, the determination unit 12 may determine that the terminal device 2 is at the designated position when a predetermined target sound or a combination of predetermined target sounds is detected from the ambient sounds.

[0065] Determination Example 3 In this example, the determination unit 12 combines Determination Example 1 and Determination Example 2. That is, in this example, an object present at a designated position is registered in advance in the information processing device 10 as a target object, and a sound present at the designated position is registered in advance in the information processing device 10 as a target sound.

[0066] A pass condition for determining that "the terminal device 2 is at the designated position" is set in advance. If the pass condition is satisfied, the determining unit 12 determines that the terminal device 2 is at the designated position.

[0067] Examples of the pass condition are those exemplified in Judgment Example 1 and Judgment Example 2. Another example of the pass condition is "person A is detected in the peripheral image, and the voice of person A is detected in the peripheral sound." Another example of the pass condition is "person A is detected in the peripheral image, and music C is detected in the peripheral sound." Another example of the pass condition is "at least one of person A is detected in the peripheral image and music C is detected in the peripheral sound is satisfied." In this way, the pass condition is defined by one object, one target sound, a combination of multiple objects, a combination of multiple target sounds, or a combination of at least one object and at least one target sound.

[0068] The definitions of "both the first object and the second object have been detected" and "both the first target sound and the second target sound have been detected" are as described above.

[0069] The above-mentioned "detection of both the first object and the first target sound" is a concept that includes at least one of the following: - The first object and the first target sound are detected simultaneously - The first target sound is detected within a predetermined time after the first object is detected - The first object is detected within a predetermined time after the first target sound is detected

[0070] 5 is a diagram illustrating an example of information pre-registered in the information processing device 10. In the illustrated example, designated position identification information, a name, a detection target, and a pass condition are associated with each other.

[0071] The "designated position identification information" is information that identifies a plurality of designated positions that have been registered in advance from one another.

[0072] "Name" is the name of each designated position.

[0073] "Detection targets" include objects and target sounds. T001 in the figure is information that distinguishes multiple objects from one another. The information in parentheses is the feature amount of the appearance of each object. S001 in the figure is information that distinguishes multiple target sounds from one another. The information in parentheses is the feature amount of each target sound.

[0074] The "pass condition" is a condition for determining that the terminal device 2 is at the designated position. In the example shown in the figure, it is indicated that if T001 is detected in the peripheral image, it is determined that the terminal device 2 is at the designated position. It is also indicated that if both T002 and T003 are detected in the peripheral image, it is determined that the terminal device 2 is at the designated position. It is also indicated that if S001 is detected in the peripheral sound, it is determined that the terminal device 2 is at the designated position.

[0075] Returning to FIG. 1, when it is determined that the terminal device 2 is not at the specified position, the error processing execution unit 13 performs error processing.

[0076] The acquisition unit 11 can continue acquiring at least one of the peripheral image and the peripheral sound while the terminal device 2 is executing the predetermined process. The determination unit 12 can continue determining whether the terminal device 2 is at the designated position while the terminal device 2 is executing the predetermined process. Under this premise, the error processing execution unit 13 may execute error processing when the determination result that the terminal device 2 is not at the designated position continues for a predetermined time or more.

[0077] As part of the error processing, the error processing execution unit 13 can output error information to the output device of the terminal device 2. The error information is composed of at least one of text, images, diagrams, pictures, and sounds.

[0078] The error information may include guidance to prompt the user to move to the specified location. For example, the error information may include a message such as "The user is not detected as being at the specified location. Please move to the specified location and perform the process."

[0079] The error information may also include guidance to take a picture of the surroundings. For example, the error information may include a message urging the user to take a picture of the surroundings, such as "The user is not detected as being at the specified position. Please take a 360-degree picture of the surroundings." Even though the terminal device 2 is at the specified position, it may happen that the target object is not detected in the surrounding image and the target sound is not detected in the surrounding audio, resulting in an erroneous determination that the terminal device 2 is not at the specified position. In such a case, by having the user take a 360-degree picture of the surroundings, the target object can be detected in the captured image, and the erroneous determination can be corrected.

[0080] Alternatively, as the error handling, the error handling execution unit 13 may stop a predetermined process being executed by the terminal device 2. The terminal device 2 stops the predetermined process in accordance with the control by the error handling execution unit 13.

[0081] If the predetermined process is an online call, the online call is disconnected (call termination) when the terminal device 2 stops the predetermined process. After completing the predetermined process, the terminal device 2 may output a message to the user such as "The call has been terminated because the user was not detected as being at the specified location."

[0082] If the predetermined process is a process of accepting input of information via a UI screen and transmitting the information to a server, when the terminal device 2 stops the predetermined process, the UI screen transitions to another screen. After completing the predetermined process, the terminal device 2 may output a message to the user such as "You were not detected to be in the specified position, so your input work cannot be accepted."

[0083] If the predetermined process is a process of outputting content to an output device and allowing the user to view it, when the terminal device 2 stops the predetermined process, the output of the content stops. For example, if audio or video is being played, the playback may stop. Then, the screen may transition to another screen. Alternatively, the screen may remain the same and the image may become hidden. After completing the predetermined process, the terminal device 2 may output a message to the user such as "You cannot view the content because we were not detected to be in the specified position."

[0084] Alternatively, as an error process, the error process execution unit 13 may first output error information to the output device of the terminal device 2. Then, if it is not determined that the terminal device 2 is at the designated position even after a predetermined time has elapsed since the output of the error information, the error process execution unit 13 may stop the predetermined process being executed by the terminal device 2. In this case, the terminal device 2 continues the predetermined process even after the error information is output. Then, the acquisition unit 11 continues to acquire at least one of the peripheral image and the peripheral sound while the terminal device 2 is executing the predetermined process. Furthermore, the determination unit 12 continues to determine whether the terminal device 2 is at the designated position while the terminal device 2 is executing the predetermined process.

[0085] 6 and 7, an example of a screen display on the terminal device 2 is shown. In this example, the predetermined process is an online call.

[0086] As shown in FIG. 6, the UI screen D displayed on the terminal device 2 includes an area Q １ and Area Q ２ Area Q １ The camera image of the person you are calling is displayed in Area Q. ２ A camera image of the user operating the terminal device 2 is displayed in the area.

[0087] Area Q １ The designated location "XX Ward Office" is displayed above the screen. The determination result of the determination unit 12 is displayed next to it. In the example of FIG. 6, "OK" is displayed. This means that it has been determined that the terminal device 2 is located at the designated location (XX Ward Office).

[0088] In this example, the acquisition unit 11 may acquire, as the peripheral image, an image generated by the terminal device 2 for transmission and reception in an online call. ２ The camera image displayed on the screen can be acquired as a peripheral image. The peripheral image can include a code R in the background of the user. １ Printed matter R displayed ２ Yadoor R ３etc. are displayed. When these objects are registered in advance as objects of interest, the determination unit 12 can determine that the terminal device 2 is at the specified position by detecting these objects from the surrounding image.

[0089] In this example, the acquisition unit 11 may acquire, as the ambient sound, the sound collected by the terminal device 2 for transmission and reception in an online call. Then, the determination unit 12 can determine that the terminal device 2 is at the specified position by detecting the target sound from such ambient sound.

[0090] In this way, when the predetermined process (an online call in the example of FIG. 6 ) uses the camera function or the microphone function to generate an image or collect an audio, the terminal device 2 can transmit the image generated by the predetermined process or the audio collected as a peripheral image or a peripheral audio to the information processing device 10. On the other hand, when the predetermined process does not use the camera function or the microphone function to generate an image or collect an audio, the terminal device 2 can separately turn on the camera function or the microphone function to acquire a peripheral image or a peripheral audio and transmit it to the information processing device 10.

[0091] Next, Fig. 7 shows an example of a UI screen D displayed on the terminal device 2 when the error processing execution unit 13 executes the error processing. In this example, error information "Please move to the NG specified position" is displayed. Area Q in Figs. 6 and 7 ２ 6, the image of the user is displayed with a code R in the background. １ Printed matter R displayed ２ Yadoor R ３ As a result, it is determined that the terminal device 2 is in the specified position. In contrast to this, in FIG. 7, the code R １ Printed matter R displayed ２ Yadoor R ３ etc. are not displayed, and instead the window R ４ As a result, it is determined that the terminal device 2 is not at the specified position.

[0092] Here, an example in which the predetermined process is an online call has been described, but the same screen display can also be used when the predetermined process is another process. For example, in area Q １ and Area Q ２ Instead of the information, information for each process (content, input UI parts, etc.) can be displayed.

[0093] Next, an example of the flow of processing by the information processing device 10 will be described using the flowchart in Fig. 8. Note that the purpose here is to explain the flow of processing. Details of each process have been described above, so a description thereof will be omitted here. Also, the example in Fig. 8 is merely an example, and can be modified as appropriate based on the above description.

[0094] First, the information processing device 10 waits for the terminal device 2 to start a predetermined process (S20). Then, when the terminal device 2 starts the predetermined process (Yes in S20), the information processing device 10 acquires at least one of a surrounding image captured by the terminal device 2 while the predetermined process is being executed and a surrounding sound collected by the terminal device 2 while the predetermined process is being executed (S21). Next, the information processing device 10 determines whether the terminal device 2 is in a specified position based on at least one of the acquired surrounding image and surrounding sound (S22).

[0095] If it is determined that the terminal device 2 is in the designated position (Yes in S23), the information processing device 10 outputs pass information (S25). As a result, for example, information such as "OK" is displayed on the UI screen D of the terminal device 2 as shown in FIG.

[0096] On the other hand, if it is determined that the terminal device 2 is not at the designated position (No in S23), the information processing device 10 outputs error information (S24). As a result, for example, information such as "NG. Please move to the designated position" is displayed on the UI screen D of the terminal device 2 as shown in FIG.

[0097] The information processing device 10 repeats the above process while the terminal device 2 continues the predetermined process (No in S26).

[0098] When the information processing device 10 determines that the terminal device 2 is at the designated position (Yes in S23), it may not execute processing such as outputting pass information (S25), but may allow the terminal device 2 to continue the predetermined processing. Only when the information processing device 10 determines that the terminal device 2 is not at the designated position (No in S23), it may execute error processing such as outputting error information (S24).

[0099] "Effects" According to the information processing device 10 of the second embodiment, the same effects as those of the information processing device 10 of the first embodiment are achieved.

[0100] Furthermore, as an error processing, the information processing device 10 can cause the output device of the terminal device 2 to output error information. As a result, the user of the terminal device 2 can be notified that an error has been detected. Then, the user can be prompted to take action to resolve the error.

[0101] Furthermore, the information processing device 10 can include in the error information a guide urging the user to move to the specified location. As a result, the user of the terminal device 2 can be notified that it has been determined that the user is not at the specified location. The information processing device 10 can then urge the user to take action to resolve the error, specifically, to move to the specified location.

[0102] Furthermore, as an error handling process, the information processing device 10 can stop a predetermined process being executed by the terminal device 2. As a result, it is possible to strictly prohibit a user from fraudulently performing a predetermined process online at a location other than the designated location.

[0103] Furthermore, the information processing device 10 can determine whether the terminal device 2 is at a designated position based on the result of a detection process that includes at least one of a process of detecting an object from a peripheral image and a process of detecting a target sound from peripheral audio. Such an information processing device 10 can accurately determine whether the terminal device 2 is at a designated position. For example, the determination can be made accurately by combining the peripheral image and the peripheral audio.

[0104] Furthermore, the information processing device 10 can make the above determination by using at least one of a person and an object present at the specified position as the target object. With such an information processing device 10, it can accurately determine whether the terminal device 2 is at the specified position.

[0105] Furthermore, the information processing device 10 can perform the above determination by using at least one of a person and an object present at the specified position as the multiple targets. With such an information processing device 10, it is possible to accurately determine whether the terminal device 2 is at the specified position.

[0106] Furthermore, the information processing device 10 can make the above determination by using a sound present at a specified position as a target sound. According to such an information processing device 10, it is possible to accurately determine whether the terminal device 2 is at a specified position.

[0107] Furthermore, the information processing device 10 can make the above determination by using multiple sounds present at the specified position as target sounds. With such an information processing device 10, it is possible to accurately determine whether the terminal device 2 is at the specified position.

[0108] Furthermore, while the terminal device 2 is executing a predetermined process, the information processing device 10 can continue to acquire at least one of a surrounding image and a surrounding sound, and determine whether the terminal device 2 is in the designated position. In other words, while the terminal device 2 is executing a predetermined process, the information processing device 10 can continue to monitor whether the terminal device 2 is in the designated position. As a result, it is possible to prevent a user from performing a predetermined process online at a location other than the designated position.

[0109] <<Third Embodiment>> In the third embodiment, a plurality of designated positions are registered in advance. Then, before operating the terminal device 2 to execute a predetermined process, the user notifies (declares) the information processing device 10 of the designated position at which the predetermined process is to be executed. The information processing device 10 determines whether the terminal device 2 is at the notified designated position based on at least one of an object and a target sound associated with the notified designated position. This will be described in detail below.

[0110] In the third embodiment, as shown in Fig. 5, a plurality of designated positions are registered in advance. At least one of an object and a sound that can be a landmark for each designated position is registered in association with each of the plurality of designated positions. The user operates the terminal device 2 while located at one of the plurality of designated positions registered in advance, and executes a predetermined process.

[0111] Before operating the terminal device 2 to execute a predetermined process, the user notifies (declares) the information processing device 10 of the designated position where the predetermined process is to be executed.

[0112] Before the terminal device 2 executes a predetermined process, the determination unit 12 identifies a designated position where the terminal device 2 executes the predetermined process, based on the notification content transmitted from the terminal device 2. Then, the determination unit 12 performs the detection process described in the second embodiment based on at least one of the target object and the target sound that are registered in association with the identified designated position.

[0113] The determination unit 12 can identify the designated position at which the terminal device 2 executes a predetermined process, for example, in any of the following specific examples 1 to 4.

[0114] Specific Example 1 After arriving at the designated location, the user performs an input to the terminal device 2 to start an operation to notify the designated location before starting a predetermined process. The terminal device 2 acquires current location information indicating the current location in response to the input. The terminal device 2 then transmits the acquired current location information to the information processing device 10. The current location information can be acquired using any widely known technology. Examples of such technology include, but are not limited to, GPS (global positioning system).

[0115] The determination unit 12 acquires the current location information transmitted from the terminal device 2. Then, the determination unit 12 compares the location information of each pre-registered designated location with the acquired current location information, and identifies a designated location that includes the location indicated by the acquired current location information. Then, the determination unit 12 recognizes the identified designated location as a designated location where the user (terminal device 2) will execute a predetermined process.

[0116] Specific Example 2 After arriving at the designated location, the user performs an input to the terminal device 2 to start an operation to notify the designated location before starting a predetermined process. In response to the input, the terminal device 2 turns on at least one of the camera function and the microphone function, and performs at least one of generating a peripheral image and collecting peripheral sound. The terminal device 2 then transmits at least one of the peripheral image and the peripheral sound to the information processing device 10. Note that the terminal device 2 may output a guidance message to the user encouraging them to take pictures of the surroundings, such as "Please take pictures of the surroundings 360 degrees."

[0117] The determination unit 12 acquires at least one of a peripheral image and a peripheral sound transmitted from the terminal device 2. Then, the determination unit 12 compares at least one of an object and a target sound at each pre-registered designated position with at least one of the acquired peripheral image and the peripheral sound, and identifies the designated position based on the comparison result. For example, based on the comparison result, the determination unit 12 identifies a designated position where at least one of the acquired peripheral image and the peripheral sound satisfies the pass condition for determining that "the terminal device 2 is at the designated position" described in the second embodiment. Then, the determination unit 12 recognizes the identified designated position as a designated position where the user (terminal device 2) will execute a predetermined process.

[0118] Specific Example 3 After arriving at the designated position, the user performs an input to the terminal device 2 to start an operation to notify the designated position before starting a predetermined process. The terminal device 2 specifies the designated position in response to the input and displays a UI screen for transmitting the specified position to the information processing device 10. The terminal device 2 then transmits the specified position specified by the user on the UI screen to the information processing device 10. The UI screen is configured to allow the user to specify one of multiple designated positions using UI components such as a pull-down list or a radio button.

[0119] The determination unit 12 acquires information indicating one designated position transmitted from the terminal device 2. Then, the determination unit 12 recognizes the one designated position as a designated position at which the user (terminal device 2) will execute a predetermined process.

[0120] Specific Example 4: At any timing before starting a predetermined process, the user performs an operation to register in advance in the information processing device 10 a designated location where the predetermined process is to be executed. The user may perform the registration operation before arriving at the designated location, or may perform the registration operation after arriving at the designated location. Furthermore, the user may perform the registration operation on the day the predetermined process is to be executed, or may perform the registration operation on a day before the day the predetermined process is to be executed.

[0121] In this example, the user installs an application dedicated to a predetermined process on the terminal device 2 and performs tasks such as membership registration. This task results in the issuance of user identification information. The user then logs in to the application dedicated to the predetermined process using the user identification information, and then performs tasks such as registering the scheduled date for executing the predetermined process and the designated location for executing the predetermined process. The information registered in this registration task is registered in the storage device of the information processing device 10.

[0122] The determination unit 12 acquires information indicating the designated location where each user plans to execute a predetermined process on that day from the information registered in advance, and recognizes the designated location indicated by the information as the designated location where each user (terminal device 2) will execute the predetermined process.

[0123] Incidentally, the determination unit 12 of the third embodiment determines whether the terminal device 2 is at the designated position based on at least one of the object and the sound associated with the designated position notified by the user as described above. In other words, the determination does not use the object and the sound associated with the designated position other than the designated position notified by the user. In this case, there is a possibility that the terminal device 2 is erroneously determined to be not at the designated position due to an incorrect notification from the user. In other words, if the user notifies that a predetermined process will be performed at designated position B despite the user actually performing a predetermined process at designated position A, the above-described erroneous determination may occur. Such inconveniences may occur when the above specific examples 3 and 4 are adopted.

[0124] Therefore, in the third embodiment, the error processing execution unit 13 may perform the following process in addition to or instead of the process described in the second embodiment. That is, the error processing execution unit 13 may output, as error information, a message to confirm whether the content of the "designated position at which the predetermined process is to be executed" notified to the information processing device 10 before starting the predetermined process is correct.

[0125] For example, the error information may include a message such as, "The XX ward office has been notified as the designated location for making an online call. Is this correct? If it is incorrect, please press the edit button to correct it." In this example, the terminal device 2 accepts an input to edit the designated location for performing a predetermined process using the edit button. The terminal device 2 then transmits the edited content to the information processing device 10. Thereafter, the determination unit 12 determines whether the terminal device 2 is at the designated location based on at least one of the target object and target sound that are registered in association with the edited designated location. The edit input may be accepted via a UI screen configured to allow the user to select one of multiple designated locations as the edited designated location using UI components such as a pull-down list or radio buttons.

[0126] Additionally, the error information may include a message prompting the user to take a picture of the surrounding area, such as, "X Ward Office has been notified as the designated location for an online call. Is this correct? Please take a 360-degree picture of the surrounding area so we can verify it." In this example, the terminal device 2 transmits at least one of the surrounding images and surrounding sounds captured and / or collected by the user in response to the guidance to the information processing device 10. The determination unit 12 identifies the designated location where the terminal device 2 is located using the same process as in the above-described identification example 2. Thereafter, the determination unit 12 determines whether the terminal device 2 is at the designated location based on at least one of the target object and target sound registered in association with the designated location identified using at least one of the surrounding images and surrounding sounds.

[0127] In addition, the error information may include a message such as, "X Ward Office has been notified as the designated location for making an online call. Is this correct? Please allow us to verify it and allow use of the location information acquisition function." In this example, when the user allows use of the location information acquisition function in response to the guidance, the terminal device 2 acquires current location information and transmits it to the information processing device 10. The determination unit 12 identifies the designated location where the terminal device 2 is located using the same process as in the above-described identification example 1. Thereafter, the determination unit 12 determines whether the terminal device 2 is at the designated location based on at least one of the target object and the target sound that are registered in association with the designated location identified using the current location information.

[0128] Other configurations of the information processing apparatus 10 of the third embodiment are similar to those of the information processing apparatus 10 of the first and second embodiments.

[0129] The information processing device 10 of the third embodiment achieves the same effects as the information processing device 10 of the first and second embodiments. Furthermore, the information processing device 10 of the third embodiment can determine whether the terminal device 2 is at a designated position based on at least one of an object and a sound that is registered in association with a designated position notified by a user among multiple designated positions. In this case, the information to be compared can be narrowed down, thereby reducing the processing load on the information processing device 10.

[0130] 9 , in a fourth embodiment, the terminal device 2 functions as the information processing device 10. In this case, the information processing device 10 (terminal device 2) acquires information on at least one of an object and a target sound that are linked to a specified position and registered in advance from the server 1. Then, the information processing device 10 (terminal device 2) executes the processing described in the first to third embodiments based on the information acquired from the server 1.

[0131] The information processing device 10 (terminal device 2) may acquire from the server 1 information on all of a plurality of pre-registered designated positions (information on at least one of the target object and the target sound).

[0132] In addition, when the configuration of the third embodiment is adopted, the information processing device 10 (terminal device 2) may obtain only information about the specified location identified in any of specific examples 1 to 4 (information about at least one of the target object and the target sound) from the server 1.

[0133] In this case, the "process of comparing the location information of each pre-registered designated location with the current location information acquired by the terminal device 2 and identifying a designated location that includes the location indicated by the acquired current location information" in specific example 1 may be performed by either the server 1 or the terminal device 2. When the amount of location information of each pre-registered designated location is enormous, the server 1 may perform this process. By having the server 1 perform this process, it becomes possible to perform the comparison without transmitting a huge amount of information to the terminal device 2.

[0134] Furthermore, the "process of comparing at least one of the target object and target sound at each designated position registered in advance with at least one of the surrounding image and surrounding sound acquired by the terminal device 2, and identifying the designated position based on the comparison result" in Identification Example 2 may be performed by either the server 1 or the terminal device 2. When there are a huge number of target objects and target sounds at each designated position registered in advance, the server 1 may perform this process. By having the server 1 perform this process, it becomes possible to perform the comparison without transmitting a huge amount of information to the terminal device 2.

[0135] Other configurations of the information processing apparatus 10 of the fourth embodiment are similar to those of the information processing apparatus 10 of the first to third embodiments.

[0136] According to the information processing device 10 of the fourth embodiment, the same effects as those of the information processing devices 10 of the first to third embodiments are realized. Furthermore, according to the fourth embodiment in which the terminal device 2 functions as the information processing device 10, the processing load on the server 1 can be reduced.

[0137] <<Fifth Embodiment>> The predetermined process of the fifth embodiment is a process performed in cooperation with another device. The predetermined process is, for example, an online call. Then, as an error process, the information processing device 10 outputs error information to at least one of the terminal device 2 executing the predetermined process and the other device. This will be described in detail below.

[0138] As shown in Fig. 10, the information processing device 10 of the fifth embodiment may be a server 1. Alternatively, as shown in Fig. 11, in the fifth embodiment, a terminal device 2 may function as the information processing device 10. Alternatively, as shown in Fig. 12, in the fifth embodiment, another device 4 may function as the information processing device 10.

[0139] As shown in FIGS. 10 to 12, a server 1, a terminal device 2, and another device 4 are connected to one another via a communication network 3 such as the Internet.

[0140] The other device 4 executes a predetermined process in cooperation with the terminal device 2. The other device 4 may be, for example, a smartphone, a tablet terminal, a personal computer, a smart watch, a mobile phone, a game console, a server, etc., but is not limited to these. The hardware configuration of such other device 4 can be realized using any widely known technology.

[0141] When the other device 4 functions as the information processing device 10 as shown in Figure 12, the information processing device 10 (the other device 4) may obtain information on all of the multiple designated locations that have been registered in advance (information on at least one of the target object and the target sound) from the server 1.

[0142] In addition, when the configuration of the third embodiment is adopted, the information processing device 10 (other device 4) may obtain only information about the specified location identified in any of specific examples 1 to 4 (information about at least one of the target object and the target sound) from the server 1.

[0143] 1 , as error processing, the error processing execution unit 13 of the information processing device 10 causes at least one of the output device of the terminal device 2 and the output device of the other device 4 to output error information. That is, in response to determining that the terminal device 2 is not at the specified position, the error processing execution unit 13 causes at least one of the output device of the terminal device 2 and the output device of the other device 4 to output error information.

[0144] As an error process, the error process execution unit 13 may output error information to both the output device of the terminal device 2 and the output device of the other device 4. The error process execution unit 13 may also cause the content of the error information output by the output device of the terminal device 2 to differ from the content of the error information output by the output device of the other device 4.

[0145] For example, the error information output by the output device of the terminal device 2 may include a guide to move to a specified location, and the error information output by the output device of the other device 4 may include a notification that the terminal device 2 is not at the specified location.

[0146] Alternatively, the error information output by the output device of the terminal device 2 may include only the determination result as to whether or not the terminal device 2 is at the designated position. The error information output by the output device of the other device 4 may indicate, in addition to the determination result, the basis for determining that the terminal device 2 is at the designated position. The basis indicates at least one of a target object and a target sound detected from at least one of a peripheral image and a peripheral sound.

[0147] Other configurations of the information processing apparatus 10 of the fifth embodiment are similar to those of the information processing apparatus 10 of the first to fourth embodiments.

[0148] The information processing device 10 of the fifth embodiment achieves the same effects as the information processing devices 10 of the first to fourth embodiments. Furthermore, when it is determined that the terminal device 2 is not at the designated location, the information processing device 10 can output error information to the output device of another device 4 that is performing a predetermined process in cooperation with the terminal device 2. Such an information processing device 10 can notify other users (users operating other devices 4) working in collaboration with the user that the user is not performing a predetermined process at the designated location. As a result, the other users can take appropriate action, such as encouraging the user to move to the designated location. Furthermore, the other users can evaluate the user based on the content of the notification.

[0149] <<Examples>> Here, examples of the information processing device 10 described in the first to fifth embodiments will be described. Note that the examples shown here are merely examples, and examples of the information processing device 10 are not limited to these.

[0150] In this embodiment, an information processing device 10 is used in the visa issuance procedure. In the visa issuance procedure, after submitting the necessary documents to a designated institution, an interview is conducted as needed. This interview is conducted only at limited locations such as embassies and consulates, which makes it inconvenient. In this embodiment, it is proposed to conduct this interview using an online call. In addition, to prevent fraud, a restriction is imposed that the interview must be conducted at a pre-designated public safe location (designated location) such as a ward office or library. Hereinafter, an interview using an online call will be referred to as an "online interview."

[0151] "Preliminary work" First, users who wish to apply for a visa prepare the necessary documents and submit them to the designated institution. Necessary documents include a letter of guarantee, a letter of invitation, a list of applicants, a company / organization profile, a schedule of stay, passport information, ID photos, and fees. Necessary documents can be submitted by mail or online. Preliminary preparation can be carried out in the same manner as before.

[0152] "Interview Reservation" When an interview becomes necessary, the user can make an appointment. The user can choose to have the interview at an embassy or consulate, or online.

[0153] First, the user installs a dedicated application for visa issuance on the terminal device 2 and logs in using newly issued user identification information. If the application has already been installed, the user logs in using the issued user identification information. After logging in, the user registers the user information and makes an appointment for the interview. If the user information has already been registered, the user only makes an appointment for the interview.

[0154] When registering user information, for example, information such as that shown in Fig. 13 is registered in the information processing device 10. In Fig. 13, user identification information, name, date of birth, address, telephone number, face photo, face information (facial feature amounts extracted from a face image), etc. are registered.

[0155] When reserving an interview, the user inputs whether they would like the interview to be held at an embassy, ​​consulate, etc., or an online interview. The user then inputs the desired interview date to reserve the interview. If an online interview is selected, the user may also input a designated location where the interview is scheduled to take place. By reserving an interview in this manner, information such as that shown in FIG. 14 is registered in the information processing device 10. FIG. 14 shows the reservation status for February 6, 2024. The person who has reserved the interview on that day and the designated location where the interview is scheduled to take place are also registered. It is not necessary to register the designated location where the interview is scheduled to take place.

[0156] Furthermore, in this embodiment, it is not possible to reserve a specific interview time. For this reason, the user only makes a reservation for the interview day. The user then makes a reservation to wait in line on the day of the interview and takes the interview when it is their turn. It is also possible to reserve a relatively wider time period, such as in the morning or afternoon. However, even in this case, the user makes a reservation to wait in line on the day of the interview and takes the interview when it is their turn. The method for making a reservation to wait in line will be described later.

[0157] Although an example of using a dedicated application for visa issuance to schedule an interview has been described here, a web page may be used instead of an application.

[0158] "On the day of the interview" Here, we will explain the process if you choose an online interview.

[0159] The user goes to the designated location on the day of the interview. If a designated location is not registered when the interview is scheduled, the user goes to one of multiple designated locations that have been notified in advance. On the other hand, if a designated location is registered when the interview is scheduled, the user goes to that designated location. Note that if the designated location registered when the interview is scheduled becomes inconvenient, the user may go to a designated location different from the one registered when the interview is scheduled.

[0160] After arriving at the designated location, the user performs an input to start an operation to notify the designated location on the terminal device 2 before starting the online interview. That is, the user launches a dedicated application for visa issuance and performs an input to start an operation to notify the designated location. In response to this operation, one of the processes in specific examples 1 to 3 described in the third embodiment is executed.

[0161] When the determination unit 12 of the information processing device 10 detects that a user is at one of the specified locations by executing any of the specific examples 1 to 3, it executes a reservation process for waiting in line for an online interview (predetermined process) in response to the detection. For example, the determination unit 12 registers information about the user in waiting information such as that shown in Figure 15. In the illustrated example, the user identification information of the waiting reservation user and the arrival time at the specified location are registered.

[0162] If the determination unit 12 cannot detect that the user is at any of the designated positions, the determination unit 12 does not execute the reservation process for waiting in line for the online interview (predetermined process). In this case, the determination unit 12 may cause the terminal device 2 to display a message urging the user to take a picture of the surroundings, such as "The user is not detected to be at the designated position. Please take a 360-degree picture of the surroundings." Then, the determination unit 12 may perform the process of detecting that the user is at any of the designated positions again based on the surrounding images and surrounding sounds acquired by the terminal device 2 in response to the message.

[0163] The determination unit 12 determines the order of the user's online interview based on the waiting list information as shown in Fig. 15. Then, when it is time to conduct the online interview, the determination unit 12 notifies the terminal device 2 of this fact.

[0164] After confirming the notification, the user operates the terminal device 2 to start the online interview. The terminal device 2 then executes the online interview (predetermined processing). The information processing device 10 then executes the processing described in the first to fifth embodiments while the online interview is being executed. That is, while the terminal device 2 (user) is executing the online interview (predetermined processing), the information processing device 10 continues to acquire at least one of a peripheral image captured by the terminal device 2 and a peripheral sound collected by the terminal device 2. The information processing device 10 then continues to determine whether the terminal device 2 (user) is at the designated position based on at least one of the acquired peripheral image and peripheral sound. If the information processing device 10 determines that the terminal device 2 (user) is not at the designated position, it performs error processing.

[0165] The information processing device 10 may also perform facial authentication of a user using images transmitted and received during an online call. The information processing device 10 may perform facial authentication using a pre-registered facial photo or facial information of the user (see FIG. 13 ). The information processing device 10 may also execute error processing when facial authentication fails. For example, the information processing device 10 may output a message indicating that facial authentication has failed to at least one of the terminal device 2 used by the user and another device 4 used by the interviewer.

[0166] Note that the user's voiceprint may be registered in advance as user information, and the information processing device 10 may perform voiceprint authentication of the user using the voice transmitted and received during online calls instead of facial authentication.

[0167] The information processing device 10 may also analyze a peripheral image while the terminal device 2 (user) is performing an online interview (predetermined processing) to detect a person not registered as a target. The information processing device 10 may also analyze peripheral sound while the terminal device 2 (user) is performing an online interview (predetermined processing) to detect the voice (voiceprint) of a person not registered as a target sound. Error processing may also be performed when the voice (voiceprint) of a person not registered as a target or a person not registered as a target sound is detected. For example, the information processing device 10 may output a notification that the presence of a third party unrelated to the online interview has been detected to at least one of the terminal device 2 used by the user and another device 4 used by the interviewer. The information processing device 10 may also output a message to the terminal device 2 used by the user encouraging the user to take the online interview alone. This configuration can prevent a user from taking an online interview while improperly receiving help from a third party.

[0168] To more effectively prevent the above-described fraud, the information processing device 10 may display a message on the terminal device 2 prompting the user to take a picture of the surroundings, such as "Please take a 360-degree picture of the surroundings," at a predetermined timing during the online interview. The information processing device 10 may then detect the above-described third party based on the surrounding images and sounds acquired by the terminal device 2 in response to the message. The information processing device 10 may repeatedly output the message and detect the third party during the online interview.

[0169] "Variation" In this example, the user conducts the online interview using their own terminal device 2. However, a terminal for taking the online interview may be placed in a pre-designated location. The user may then use that terminal to take the online interview. If the terminal is portable, the user may take the terminal to another location and take the online interview there. The information processing device 10 can detect such fraud.

[0170] Furthermore, the information processing device 10 may link the determination result by the determination unit 12 to the user identification information of the user taking the online interview and register it as log information. With this configuration, the determination result by the determination unit 12 can be reviewed at a later date.

[0171] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0172] In addition, in the flowcharts used in the above description, multiple steps (processes) are described in order. However, the order of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the drawings can be changed to the extent that the content is not affected. Furthermore, the above-described embodiments can be combined to the extent that the content is not contradictory.

[0173] Some or all of the above embodiments may be described as in the following supplementary notes, but are not limited to them. 1. An information processing device having: acquisition means for acquiring at least one of a peripheral image captured by a terminal device executing a predetermined process and a peripheral sound collected by the terminal device; determination means for determining whether the terminal device is at a designated position based on at least one of the peripheral image and the peripheral sound; and error processing execution means for performing error processing when it is determined that the terminal device is not at the designated position. 2. The information processing device described in 1, wherein the error processing execution means causes an output device of the terminal device to output error information as the error processing. 3. The information processing device described in 2, wherein the error information includes guidance urging the user to move to the designated position. 4. The information processing device described in any of 1 to 3, wherein the predetermined process is a process performed in cooperation with another device, and the error processing execution means causes at least one of an output device of the terminal device and an output device of the other device to output error information as the error processing. 5. 5. The information processing device according to 4, wherein the error processing execution means causes both the output device of the terminal device and the output device of the other device to output the error information as the error processing, and causes the content of the error information output by the output device of the terminal device to differ from the content of the error information output by the output device of the other device. 6. The information processing device according to 5, wherein the error information output by the output device of the terminal device includes guidance urging the user to move to the designated position, and the error information output by the output device of the other device includes a notification that the terminal device is not at the designated position. 7. The information processing device according to 1, wherein the error processing execution means stops the predetermined process being executed as the error processing. 8. The information processing device according to any of 1 to 7, wherein the determination means determines whether the terminal device is at the designated position based on a result of a detection process including at least one of a process of detecting an object from the peripheral image and a process of detecting a target sound from the peripheral audio.9. The information processing device according to 8, wherein at least one of a person and an object present at the designated position is registered in advance as the target object, and the determination means determines that the terminal device is at the designated position when the target object is detected in the peripheral image. 10. The information processing device according to 9, wherein at least one of a person and an object present at the designated position is registered in advance as a plurality of the targets, and the determination means determines that the terminal device is at the designated position when a predetermined one of the targets or a predetermined combination of a plurality of the targets is detected in the peripheral image. 11. The information processing device according to any one of 8 to 10, wherein a sound present at the designated position is registered in advance as the target sound, and the determination means determines that the terminal device is at the designated position when the target sound is detected in the peripheral sound. 12. The information processing device according to 11, wherein a plurality of sounds present at the designated position are registered in advance as the target sound, and the determination means determines that the terminal device is at the designated position when a predetermined one of the target sounds or a predetermined combination of a plurality of the target sounds is detected in the peripheral sound. 13. 14. The information processing device according to any one of 8 to 12, wherein there are a plurality of the designated positions, and at least one of the object and the target sound is registered in advance and linked to each of the plurality of designated positions, and the determination means identifies the designated position at which the terminal device will execute the predetermined process before the terminal device starts the predetermined process, and performs the detection process based on at least one of the object and the target sound that is registered and linked to the identified designated position. 14. The information processing device according to 13, wherein the determination means identifies the designated position at which the terminal device will execute the predetermined process based on at least one of the surrounding image captured by the terminal device and the surrounding sound collected by the terminal device before starting the predetermined process. 15. The information processing device according to any one of 1 to 14, wherein while the terminal device is executing the predetermined process, the acquisition means continues to acquire at least one of the surrounding image and the surrounding sound, and the determination means continues to determine whether the terminal device is at the designated position.16. An information processing device according to any one of 1 to 15, wherein the predetermined process is a process performed in cooperation with another device, and the determination means detects that the terminal device is at the designated position before the terminal device starts the predetermined process, and executes reservation processing for waiting its turn for the predetermined process in response to detection that the terminal device is at the designated position. 17. An information processing device according to any one of 1 to 16, wherein the predetermined process is a process that is requested to be executed at the designated position. 18. An information processing device according to any one of 1 to 17, wherein the predetermined process is an online call. 19. An information processing method in which one or more computers acquire at least one of a peripheral image captured by a terminal device executing a predetermined process and a peripheral sound collected by the terminal device, determine whether the terminal device is at the designated position based on at least one of the peripheral image and the peripheral sound, and perform error processing if it is determined that the terminal device is not at the designated position. 20. A recording medium for recording a program that causes a computer to function as: an acquisition means for acquiring at least one of a peripheral image captured by a terminal device executing a predetermined process and a peripheral sound collected by the terminal device; a determination means for determining whether the terminal device is at a specified position based on at least one of the peripheral image and the peripheral sound; and an error processing execution means for performing error processing when it is determined that the terminal device is not at the specified position.

[0174] Some or all of Supplements 2 to 18 that are dependent on the information processing device of Supplement 1 described above may also be dependent on the information processing method of Supplement 19 and the recording medium of Supplement 20 in the same dependent relationship as Supplement 1 and Supplements 2 to 18. Furthermore, within the scope of each of the above-mentioned embodiments, some or all of the configurations described as Supplements can be realized in various hardware, software, various recording means for recording software, or systems.

[0175] REFERENCE SIGNS LIST 1 Server 2 Terminal device 3 Communication network 4 Other device 10 Information processing device 11 Acquisition unit 12 Determination unit 13 Error processing execution unit 1A Processor 2A Memory 3A Input / output I / F 4A Peripheral circuit 5A Bus

Claims

1. An information processing device having: an acquisition means for acquiring at least one of a peripheral image captured by a terminal device executing a predetermined process and a peripheral sound collected by the terminal device; a determination means for determining whether the terminal device is at a specified position based on at least one of the peripheral image and the peripheral sound; and an error processing execution means for performing error processing when it is determined that the terminal device is not at the specified position.

2. The information processing device according to claim 1, wherein the error processing execution means causes an output device of the terminal device to output error information as the error processing.

3. The information processing device according to claim 2, wherein the error information includes a guide to urge the user to move to the specified position.

4. An information processing device according to any one of claims 1 to 3, wherein the specified processing is processing performed in cooperation with another device, and the error processing execution means causes error information to be output to at least one of an output device of the terminal device and an output device of the other device as the error processing.

5. The information processing device according to claim 4, wherein the error processing execution means outputs the error information to both the output device of the terminal device and the output device of the other device as the error processing, and differentiates the content of the error information output by the output device of the terminal device from the content of the error information output by the output device of the other device.

6. An information processing device as described in claim 5, wherein the error information output by the output device of the terminal device includes guidance urging the user to move to the specified location, and the error information output by the output device of the other device includes a notification that the terminal device is not at the specified location.

7. The information processing device according to claim 1, wherein the error processing execution means stops the predetermined processing being executed as the error processing.

8. An information processing device according to any one of claims 1 to 7, wherein the determination means determines whether the terminal device is at the specified position based on the results of a detection process including at least one of a process of detecting an object from the surrounding image and a process of detecting a target sound from the surrounding audio.

9. An information processing device as described in claim 8, wherein at least one of a person and an object present at the specified position is registered in advance as the target object, and the determination means determines that the terminal device is at the specified position when the target object is detected from the surrounding image.

10. An information processing device as described in claim 9, wherein at least one of a person and an object present at the specified location is registered in advance as a plurality of the objects, and the determination means determines that the terminal device is at the specified location when a predetermined one of the objects or a combination of a predetermined plurality of the objects is detected from the surrounding image.

11. An information processing device according to any one of claims 8 to 10, wherein a sound present at the designated position is registered in advance as the target sound, and the determination means determines that the terminal device is at the designated position when the target sound is detected from the surrounding sounds.

12. An information processing device as described in claim 11, wherein a plurality of sounds present at the designated position are registered in advance as the target sounds, and the determination means determines that the terminal device is at the designated position when a predetermined target sound or a combination of a predetermined plurality of target sounds is detected from the surrounding sounds.

13. An information processing device according to any one of claims 8 to 12, wherein there are a plurality of designated positions, and at least one of the target object and the target sound is registered in advance in association with each of the plurality of designated positions, and the determination means identifies the designated position at which the terminal device will execute the predetermined processing before the terminal device starts the predetermined processing, and performs the detection processing based on at least one of the target object and the target sound registered in association with the identified designated position.

14. An information processing device as described in claim 13, wherein the determination means identifies the designated position at which the terminal device executes the specified processing based on at least one of the surrounding image captured by the terminal device and the surrounding sound collected before starting the specified processing.

15. An information processing device described in any one of claims 1 to 14, wherein while the terminal device is executing the specified processing, the acquisition means continues to acquire at least one of the surrounding image and the surrounding sound, and the determination means continues to determine whether the terminal device is at the specified position.

16. An information processing device as described in any one of claims 1 to 15, wherein the specified processing is processing performed in cooperation with another device, and the determination means detects that the terminal device is at the designated location before the terminal device starts the specified processing, and executes a reservation process for waiting in line for the specified processing in response to the detection that the terminal device is at the designated location.

17. An information processing device according to any one of claims 1 to 16, wherein the predetermined process is a process that is requested to be executed at the specified position.

18. An information processing device according to any one of claims 1 to 17, wherein the predetermined process is an online call.

19. An information processing method in which one or more computers acquire at least one of a surrounding image captured by a terminal device executing a specified process and a surrounding sound collected by the terminal device, determine whether the terminal device is in a specified position based on at least one of the surrounding image and the surrounding sound, and perform error processing if it is determined that the terminal device is not in the specified position.

20. A recording medium that records a program that causes a computer to function as: an acquisition means that acquires at least one of a peripheral image captured by a terminal device executing a specified process and a peripheral sound collected by the terminal device; a determination means that determines whether the terminal device is at a specified position based on at least one of the peripheral image and the peripheral sound; and an error processing execution means that performs error processing when it is determined that the terminal device is not at the specified position.

Images