Voice input device, voice input system and voice input method

Abstract

To provide a voice input device, a voice input system and a voice input method, which can more effectively suppress impersonation by others in reception of voice input.SOLUTION: In a voice input system, a control terminal S1 being a voice input device includes: a communication part 20 being an acquisition part for acquiring voice data of a worker performing work; a sentence interpretation part 212 for voice-recognizing voice data and performing sentence interpretation on a recognized utterance content; a voiceprint authentication part 214B being a determination part for determining whether uttered voice corresponding to a sentence interpretation result in voice data is uttered voice of the worker or not based on the sentence interpretation result of the utterance content; and an action determination part 215 being a control part for receiving input operation corresponding to the sentence interpretation result based on the sentence interpretation result and whether uttered voice corresponding to the sentence interpretation result is uttered voice of the worker.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a voice input device, a voice input system, and a voice input method. [Background technology]

[0002] Patent Document 1 discloses an authentication device that authenticates a speaker based on the input voice of the speaker. The authentication device saves the speech pattern used for speaker registration, assigns to the speaker a speech pattern for speaker authentication that includes at least one set of a sequence of multiple words included in the saved speaker registration pattern, and judges whether the speaker is the registered speaker based on a pre-registered feature amount and a feature amount extracted from the input voice. By configuring a speaker authentication pattern that includes at least one set of a sequence of multiple words, the authentication device can prevent false identity fraud by recording and playing back the registered speaker's voice, while maintaining higher authentication accuracy than when the speech pattern for speaker authentication is random. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2004-62076 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, Patent Document 1 is intended to prevent identity fraud (i.e., spoofing) during identity authentication, and does not anticipate preventing identity fraud (i.e., spoofing) by others when continuing to accept the speaker's voice input after identity authentication.

[0005] The present disclosure has been devised in consideration of the above-described conventional situation, and aims to provide a voice input device, a voice input system, and a voice input method that can more effectively prevent impersonation by others when accepting voice input.

[0006] The present disclosure provides a voice input device including an acquisition unit that acquires voice data of a worker performing work, a sentence interpretation unit that performs voice recognition on the voice data and performs sentence interpretation on the recognized speech content, a determination unit that determines whether or not a speech voice in the voice data that corresponds to the sentence interpretation result is the speech voice of the worker based on the sentence interpretation result of the speech content, and a control unit that accepts an input operation corresponding to the sentence interpretation result based on the sentence interpretation result and whether or not the speech voice corresponding to the sentence interpretation result is the speech voice of the worker.

[0007] The present disclosure also provides a voice input system including a terminal device that collects the voice of a worker performing work, and a control terminal capable of communicating with the terminal device, wherein the terminal device transmits voice data of the worker to the terminal device, and the control terminal acquires the voice data, performs voice recognition on the voice data, performs sentence interpretation on the recognized speech content, and determines whether or not a speech voice in the voice data that corresponds to the sentence interpretation result is the speech voice of the worker based on the sentence interpretation result and whether or not the speech voice corresponding to the sentence interpretation result is the speech voice of the worker, and accepts an input operation corresponding to the sentence interpretation result based on the sentence interpretation result and whether or not the speech voice corresponding to the sentence interpretation result is the speech voice of the worker.

[0008] The present disclosure also provides a voice input method performed by a computer, which acquires voice data of a worker performing work, performs voice recognition on the voice data, performs sentence interpretation on the recognized speech content, and determines whether or not a speech voice in the voice data that corresponds to the sentence interpretation result is the speech voice of the worker based on the sentence interpretation result, and accepts an input operation corresponding to the sentence interpretation result based on the sentence interpretation result and whether or not the speech voice corresponding to the sentence interpretation result is the speech voice of the worker. Effect of the Invention

[0009] According to the present disclosure, impersonation by others can be more effectively prevented when accepting voice input. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of the overall configuration of a voice input system according to a first embodiment; [Diagram 2] FIG. 1 is a block diagram showing an example of the internal configuration of a terminal device and a control terminal according to a first embodiment; [Diagram 3] FIG. 1 is a diagram showing an example of voiceprint registration in the voice input system according to the first embodiment. [Figure 4] FIG. 13 is a diagram showing an example of operation transition of a control terminal in the first embodiment. [Diagram 5] A flowchart for explaining an example of an operation procedure in an initial state of a control terminal in the first embodiment. [Figure 6] A flowchart for explaining an example of an operation procedure of the control terminal in the cyclic mode in the first embodiment. [Figure 7] A flowchart for explaining an example of a procedure for switching the control terminal from a cyclic mode to another mode in the first embodiment. [Figure 8] A flowchart for explaining an example of an operation procedure of a control terminal in a suspend mode according to the first embodiment. [Figure 9] A flowchart for explaining an example of an operation procedure of the control terminal in the standby mode according to the first embodiment. [Figure 10] FIG. 13 is a diagram showing an example of the overall configuration of a voice input system according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] (Background to this disclosure) In recent years, digital transformation (DX) using voice input has been progressing, for example, in security or patrols, factory inspections, etc. For example, there is an application that supports security patrols by accepting voice input of patrol work records by patrol personnel and recording the voice-input patrol work content. Such an application allows patrol personnel to record patrol work without directly touching a terminal device during patrol, and also makes it easy to refer to or analyze the recorded patrol work content.

[0012] However, patrols are sometimes performed in an environment where there are people other than the patrol personnel. Therefore, when using the above-mentioned application, there is a demand for a technology that can distinguish between the voice of the patrol personnel and other voices and prevent impersonation by people other than the patrol personnel.

[0013] Also, there is a conventional personal authentication device that can prevent identity fraud (i.e., spoofing) by recording and playing back the voice of a registered person by configuring a speaker authentication pattern that includes at least one concatenation of multiple words, while maintaining higher authentication accuracy than if the speech pattern for speaker authentication were random (Patent Document 1). However, since Patent Document 1 prevents identity fraud (i.e., spoofing) by limiting the speaker authentication pattern to the speech pattern used for speaker registration, applying this to the personal authentication when inputting the voice of the above-mentioned patrol work record was very time-consuming because all the speech patterns related to the patrol work record had to be registered in advance.

[0014] Hereinafter, with reference to the drawings as appropriate, a detailed description will be given of an embodiment specifically disclosing a voice input device, a voice input system, and a voice input method according to the present disclosure. However, more detailed description than necessary may be omitted. For example, detailed description of already well-known matters and duplicated description of substantially the same configuration may be omitted. This is to avoid the following description becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0015] First, a voice input system 100 according to the first embodiment will be described with reference to Fig. 1 and Fig. 2. Fig. 1 is a diagram showing an example of the overall configuration of the voice input system 100 according to the first embodiment. Fig. 2 is a block diagram showing an example of the internal configuration of each of the terminal device P1 and the control terminal S1 in the first embodiment. It goes without saying that the voice input system 100 shown in Fig. 1 is only an example and is not limited to this. Also, in Fig. 2, the network NW and the monitor MN are omitted from illustration.

[0016] The voice input system 100 accepts a voice input operation based on the speech of a patrol officer (e.g., a security guard, police, manager, or worker) during patrol work such as security or patrol rounds, or factory inspections. The voice input system 100 performs identity verification (impersonation detection) of the patrol officer based on the voice data, and performs work support for the patrol officer based on the voice input operation based on the identity verification result.

[0017] 1, when the voice input system 100 receives an input of a voice INP "1st floor warehouse, no abnormalities" from the patrol officer INS, the control terminal S1 records the patrol location "1st floor warehouse" and the patrol work result "no abnormalities" in association with each other based on the voice INP "1st floor warehouse, no abnormalities". In addition, when the voice input system 100 completes recording of the patrol work result by the control terminal S1, it generates a synthetic voice ANS instructing the next patrol location, "No abnormalities, recorded, please move on to inspecting the windows on the 2nd floor", and notifies the patrol officer INS.

[0018] The voice input system 100 includes a terminal device P1, a network NW, a control terminal S1, and a monitor MN. The monitor MN may be omitted from the voice input system 100. The voice input system 100 may include a plurality of terminal devices and a plurality of control terminals.

[0019] The terminal device P1 is a terminal device carried by a patrol worker INS during the patrol work. The terminal device P1 is connected to the control terminal S1 via the network NW so as to be able to transmit and receive data, and realizes a UI display function based on a control command transmitted from the control terminal S1.

[0020] The terminal device P1 is realized by, for example, a smartphone, a tablet terminal, or the like. The terminal device P1 includes a communication unit 10, a processor 11, a memory 12, and a display unit 13. The terminal device P1 may include an audio input device DV1 or an audio output device DV2. The audio input device DV1 and the audio output device DV2 may each be realized by a single device such as a headset, or may each be realized by different devices such as a microphone and earphones.

[0021] The communication unit 10 is connected to the control terminal S1 via the network NW so as to be capable of wireless communication, and executes transmission and reception of data. Note that the wireless communication here refers to communication via a wireless Local Area Network (LAN) such as Wi-Fi (registered trademark).

[0022] The communication unit 10 outputs various data transmitted from the control terminal S1 to the processor 11. In addition, the communication unit 10 transmits various data output from the processor 11 to the control terminal S1 via the network NW.

[0023] The processor 11 is configured using, for example, a Central Processing Unit (hereinafter, referred to as "CPU") or a Field Programmable Gate Array (hereinafter, referred to as "FPGA"), and performs various processes and controls in cooperation with the memory 12. Specifically, the processor 11 refers to the programs and data stored in the memory 12, and executes the programs to realize the respective functions of the audio input unit 111 and the audio output unit 112, and a UI display function.

[0024] The processor 11 generates a UI screen (not shown) based on the control command transmitted from the control terminal S1, and outputs and displays it on the display unit 13. The UI screen referred to here is a screen capable of supporting the patrol work performed by the patrol person INS, and may be, for example, a screen indicating the patrol route, a screen indicating the patrol locations, a screen indicating the patrol order, a screen including the patrol work history, and the like.

[0025] The voice input unit 111 outputs voice data input from the voice input device DV1 to the communication unit 10 and transmits it to the control terminal S1. The voice input unit 111 may output the voice data in association with identification information (e.g., a serial number, a management number, an ID, etc.) that is assigned to the terminal device P1 and can identify the terminal device P1.

[0026] The voice output unit 112 outputs the voice synthesis signal or voice data transmitted from the control terminal S1 to the voice output device DV2. The voice synthesis signal will be described later.

[0027] The memory 12 has, for example, a Random Access Memory (hereinafter, referred to as "RAM") as a work memory used when executing each process of the processor 11, and a Read Only Memory (hereinafter, referred to as "ROM") for storing programs and data that define the operation of the processor 11. The RAM temporarily stores data or information generated or acquired by the processor 11. The ROM has written therein programs that define the operation of the processor 11.

[0028] The display unit 13 is configured using a display such as a Liquid Crystal Display (hereinafter, referred to as "LCD"), an organic electroluminescence (hereinafter, referred to as "EL"), a touch panel, etc. The display unit 13 displays various UI screens (not shown) output from the processor 11.

[0029] The voice input device DV1 is realized by a device capable of collecting the speech of the patrolman INS, such as a microphone or a headset equipped with a microphone. The voice input device DV1 converts the collected speech into voice data and outputs it to the voice input unit 111. The voice input device DV1 may be provided in the terminal device P1. The voice input device DV1 may also output the voice data to the voice input unit 111 by wired communication using a microphone jack or a Universal Serial Bus (USB) or the like, or by wireless communication.

[0030] The audio output device DV2 is realized by a device capable of outputting audio, such as a speaker or a headset. The audio output device DV2 converts the synthesized voice signal or audio data output from the audio output unit 112 into audio and outputs it. The audio output device DV2 may be provided in the terminal device P1. The audio output device DV2 is connected to the audio output unit 112 by wired communication using a microphone jack or a USB, or wireless communication.

[0031] The control terminal S1 (an example of a computer) is realized by a personal computer (hereinafter referred to as "PC"), a notebook PC, a tablet terminal, a smartphone, a cloud server, or the like installed in a monitoring center that monitors or manages the patrol work performed by at least one patrol staff member INS. The control terminal S1 realizes functions such as voice recognition, voiceprint authentication, voiceprint registration, behavior recording, feature extraction, and behavior recording based on the voice data transmitted from the terminal device P1.

[0032] The control terminal S1 is connected to at least one terminal device P1 so as to be able to transmit and receive data therebetween. The control terminal S1 includes a communication unit 20, a processor 21, a memory 22, and a voiceprint registration database DB. Note that the voiceprint registration database DB may be configured separately from the control terminal S1.

[0033] The communication unit 20 (an example of an acquisition unit) is connected to at least one terminal device P1 via the network NW so as to be capable of wireless communication, and executes transmission and reception of data. The communication unit 20 outputs various data transmitted from the terminal device P1 to the processor 21. The communication unit 20 also transmits various data output from the processor 21 to the terminal device P1 via the network NW.

[0034] The processor 21 is configured using, for example, a CPU or an FPGA, and performs various processes and controls in cooperation with the memory 22. Specifically, the processor 21 refers to the programs and data stored in the memory 22 and executes the programs to realize the functions of each unit and various functions. The units referred to here are a voice recognition unit 211, a sentence interpretation unit 212, a feature extraction unit 213, a voiceprint processing unit 214, an action determination unit 215, a voice synthesis unit 216, a report communication unit 217, and a behavior recording unit 218, etc. The various functions referred to here are functions such as voice recognition, voiceprint authentication, voiceprint registration, behavior recording, feature extraction, and behavior recording.

[0035] The voice recognition unit 211 executes voice recognition based on the voice data transmitted from the terminal device P1. The voice recognition unit 211 converts the spoken content based on the recognized voice recognition result into text data, and outputs it to the sentence interpretation unit 212. The voice recognition unit 211 may estimate the emotion of the patrol officer INS based on the voice data, and output the estimated emotion information to the sentence interpretation unit 212 in association with the text data.

[0036] The sentence interpretation unit 212 executes sentence interpretation based on the text data output from the speech recognition unit 211, generates metadata corresponding to the sentence interpretation result, and outputs it to the action determination unit 215. Note that the sentence interpretation process may use any known technology such as natural language processing.

[0037] The feature extraction unit 213 extracts features indicating the individuality of the speaker from the voice data transmitted from the terminal device P1. The feature extraction unit 213 outputs metadata indicating the extracted features to the voiceprint processing unit 214.

[0038] Voiceprint processing unit 214 executes voiceprint registration or voiceprint authentication based on the metadata output from feature amount extraction unit 213. Voiceprint processing unit 214 realizes the functions of voiceprint registration unit 214A and voiceprint authentication unit 214B.

[0039] Voiceprint processing unit 214 executes voiceprint registration by voiceprint registration unit 214A when a control command requesting voiceprint registration is associated with the voice data transmitted from terminal device P1. Voiceprint processing unit 214 executes voiceprint authentication by voiceprint authentication unit 214B when a control command requesting voiceprint registration is not associated with the voice data transmitted from terminal device P1.

[0040] The voiceprint registration unit 214A associates the voiceprint of the patrol officer INS with information about the patrol officer INS (for example, information that can identify the patrol officer INS, such as name and employee ID) and registers it in the voiceprint registration database DB based on the metadata output from the feature extraction unit 213. Note that the processor 21 may acquire information about the patrol officer INS by performing sentence analysis on the contents of the spoken voice based on the voice data, or may acquire information about the patrol officer INS from the terminal device P1.

[0041] Voiceprint authentication unit 214B (an example of a determination unit) executes voiceprint authentication based on the metadata output from feature extraction unit 213. Specifically, voiceprint authentication unit 214B determines whether or not patrol officer INS is a person registered in voiceprint registration database DB based on the metadata output from feature extraction unit 213 and voiceprint data (feature) of at least one person registered in voiceprint registration database DB. Voiceprint authentication unit 214B outputs the voiceprint authentication result to action determination unit 215.

[0042] Furthermore, the voiceprint authentication unit 214B executes speaker matching or speaker identification based on the voice data or voiceprint of the patrol officer INS authenticated by voiceprint authentication and the voice data transmitted from the terminal device P1. The voiceprint authentication unit 214B outputs the result of the speaker matching or speaker identification to the action determination unit 215.

[0043] The action determination unit 215 (an example of a voice input device) determines a support operation (hereinafter, referred to as "action") to be performed on the patrol officer INS based on the metadata (sentence interpretation result) output from the sentence interpretation unit 212 and the voiceprint authentication result output from the voiceprint authentication unit 214B. Based on the determined action, the action determination unit 215 generates text data for the patrol officer INS to execute instructions (support) by voice guide, generates metadata for reporting to the police or an external party such as a manager, and generates metadata including the results of the patrol work by the patrol officer INS. The action determination unit 215 outputs the generated various data to each unit (voice synthesis unit 216, report communication unit 217, or behavior recording unit 218) corresponding to the determined action.

[0044] The voice synthesis unit 216 generates voice synthesis data by synthesizing a voice for giving instructions (assistance) to the patrol officer INS by voice guidance based on the text data output from the action determination unit 215. The voice synthesis unit 216 transmits the generated voice synthesis data to the terminal device P1.

[0045] The report communication unit 217 issues a report to a previously set report destination based on the metadata output from the action determination unit 215, and starts a call between the report destination and the terminal device P1. Note that the report destination here may be set arbitrarily, and may be, for example, a police station, a manager of a monitoring center, or the like.

[0046] The behavior recording unit 218 records the patrol work results by the patrol person INS based on the metadata output from the action determination unit 215. The behavior recording unit 218 records information about the patrol person INS who is a patrol person, the patrol work results, patrol work location information, and patrol date and time information in association with each other.

[0047] The memory 22 has, for example, a RAM as a work memory used when executing each process of the processor 21, and a ROM for storing programs and data that define the operation of the processor 21. The RAM temporarily stores data or information generated or acquired by the processor 21. The ROM has written therein programs that define the operation of the processor 21.

[0048] The voiceprint registration database DB is a so-called storage device and is configured using a storage medium such as a flash memory, a hard disk drive (hereinafter referred to as "HDD"), a solid state drive (hereinafter referred to as "SSD"), etc. The voiceprint registration database DB stores at least one patrol officer INS and a voiceprint indicating the features of the speech of the patrol officer INS in association with each other.

[0049] The monitor MN is configured using a display such as an LCD, an organic EL, or a touch panel. The monitor MN displays various screens (not shown) output from the processor 21. The various screens referred to here are screens including information about the patrolman INS who has been voiceprint authenticated or whose voiceprint has been registered, screens including location information of the patrolman INS at the time of the report, etc. The monitor MN is not essential and may be omitted.

[0050] Next, an example of voiceprint registration will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of voiceprint registration in the speech input system 100 according to the first embodiment. Note that the voiceprint registration example shown in Fig. 3 is merely an example and is not intended to be limiting. For example, information about the tour staff INS may be acquired by the tour staff INS himself inputting information about the tour staff INS to the terminal device P1. Furthermore, voiceprint registration of the tour staff INS does not have to be performed based on the sentence interpretation result.

[0051] In the example shown in Figure 3, the terminal device P1 transmits voice data of the spoken voice COM1 of the patrol officer INS, "Registering the voice. Name is ○○××, employee ID is 1234567. Test voice for registration. No abnormalities at point A. Suspicious baggage found at point B. Requesting instructions." to the control terminal S1.

[0052] The control terminal S1 extracts features indicating the individuality of the patrol officer INS from the voice data of the spoken voice COM1. The control terminal S1 also performs sentence interpretation on the voice data of the spoken voice COM1 and obtains the name "XXX" and employee ID "1234567", which are information about the patrol officer INS, from the contents of the spoken voice COM1. The control terminal S1 associates the extracted features of the spoken voice (voiceprint data) with information about the patrol officer INS and registers them in the voiceprint registration database DB, and also generates a UI screen including the registered contents and transmits it to the terminal device P1.

[0053] The terminal device P1 outputs and displays the UI screen including the registration contents transmitted from the control terminal S1 on the display unit 13. The terminal device P1 accepts the displayed UI screen, i.e., the input of the spoken voice COM2 "OK" by the patrol officer INS regarding the registration contents, and transmits the voice data of the inputted spoken voice COM2 to the control terminal S1.

[0054] As described above, the voice input system 100 according to the first embodiment can accept voice-based voiceprint registration of the patrol officer INS.

[0055] Next, the operation transition of the control terminal S1 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the operation transition of the control terminal S1 in the first embodiment.

[0056] The control terminal S1 transitions to one of a number of operation modes based on the voice data transmitted from the terminal device P1 or the presence or absence of voice input, and supports the patrol work performed by the patrol person INS.

[0057] After completing the initial state process (step St100) for identifying the patrol person INS and supporting the start of patrol work by the patrol person INS, the control terminal S1 transitions to a patrol mode (step St200) for supporting the patrol person INS during patrol work. In addition, the control terminal S1 transitions the operation mode among four modes based on the voice input operation by the patrol person INS: a patrol mode (step St200) for supporting the patrol work by the patrol person INS, an interruption mode (step St300) for accepting interruption of the patrol work by the patrol person INS, a standby mode (step St400) for waiting for the resumption of the patrol work by the patrol person INS, and a report mode for reporting an abnormality or trouble occurring during patrol work to the outside (for example, the police, a monitoring center, etc.).

[0058] The terminal device P1 accepts an operation by the patrol person INS to complete the patrol work and transmits it to the control terminal S1. Based on the notification of patrol work completion transmitted from the terminal device P1, the control terminal S1 generates a report (not shown) including the results of the patrol work performed by the patrol person INS and transmits it to the terminal device P1. The terminal device P1 outputs and displays the report transmitted from the control terminal S1 on the display unit 13, and when the patrol person INS has completed visual confirmation of the report, it terminates the application and ends the patrol work support.

[0059] Each mode will be explained below.

[0060] <Cycling mode> The patrol mode is a mode in which the patrol officer INS is assisted in patrol work by synthetic voice or UI screens, etc.

[0061] In the patrol mode, the control terminal S1 constantly records the voice data transmitted from the terminal device P1. The control terminal S1 generates voice synthesis or a UI display screen including instructions related to the patrol work of the patrol officer INS, such as the patrol route and patrol locations, and transmits it to the terminal device P1. The terminal device P1 supports the patrol work of the patrol officer INS by outputting the voice synthesis or UI display screen transmitted from the control terminal S1.

[0062] The control terminal S1 also performs voice recognition and sentence interpretation on the voice data transmitted from the terminal device P1, and obtains and records the patrol work result for the patrol work location (for example, "no abnormality", "abnormality present", etc.) from the sentence interpretation result. The control terminal S1 generates voice synthesis or a UI display screen including instructions related to the patrol work such as the next patrol location and the patrol route to the next patrol location, and transmits it to the terminal device P1.

[0063] <Suspend mode> The interruption mode is executed when the patrolman INS temporarily stops patrolling and does not wish to record the contents of the conversation.

[0064] In the interruption mode, the control terminal S1 omits recording of voice data transmitted from the terminal device P1 and omits prompting of voice input to the patrol officer INS while the interruption mode continues. In addition, the control terminal S1 performs voice recognition and sentence interpretation on the voice data transmitted from the terminal device P1, and in accepting voice input operations based on the results of sentence interpretation, accepts only voice input operations to switch from the interruption mode to the patrol mode (for example, "return to patrol mode", "return to patrol route", etc.).

[0065] <Standby mode> The standby mode is a mode that is executed when the patrol officer INS makes a speech that is unrelated to the results of the patrol work, such as when the patrol officer INS discovers a suspicious person during patrol work and attempts to communicate with the suspicious person, or when the patrol officer INS is called out to by a third party, and when the person requests that the speech be recorded.

[0066] In the standby mode, the control terminal S1 omits prompting the patrol officer INS for voice input and continues recording the voice data transmitted from the terminal device P1. The control terminal S1 also performs voice recognition and sentence interpretation on the voice data transmitted from the terminal device P1, and in accepting a voice input operation based on the sentence interpretation result, accepts a voice input operation to switch from the standby mode to the patrol mode (e.g., "return to patrol mode," "return to patrol route," etc.) and a voice input operation to request a report (e.g., "emergency occurrence," "report," "please call," etc.). The control terminal S1 switches from the standby mode to the patrol mode or to the report mode based on the voice input operation.

[0067] <Report mode> The report mode is a mode that is executed when the patrol officer INS makes any utterance unrelated to the results of the patrol, such as when the patrol officer INS discovers a suspicious person while on patrol and attempts to communicate with the suspicious person, or when the patrol officer INS is asked for directions by a third party, and when a report is requested.

[0068] In the report mode, the control terminal S1 records the voice data transmitted from the terminal device P1, and continuously executes speech recognition of the voice data, conversion processing to text data, and sentence interpretation processing based on the text data. The control terminal S1 determines the likelihood that the speaker of the voice corresponding to the specific word or intention that triggered the transition from another mode to the report mode is the voice of the patrol officer INS. Based on the result of this determination, the control terminal S1 determines the importance of the report, and based on the determined importance of the report, determines the report waiting time before reporting to the report destination.

[0069] The control terminal S1 executes a report to a previously registered report destination when the report waiting time has elapsed, and establishes a connection between the previously set report destination and the terminal device P1 so that communication can be performed between them. When the control terminal S1 determines that the communication between the report destination and the terminal device P1 has ended, or based on a voice or manual input operation by the patrol officer INS, the control terminal S1 transitions to a patrol mode. The control terminal S1 may transmit to the report destination information as to whether or not the speaker of the voice corresponding to the specific word or intention that triggered the transition to the report mode before the report is the voice of the patrol officer INS.

[0070] For example, when the control terminal S1 determines that the speaker of the voice corresponding to the specific word or intention that triggered the transition to the report mode is the voice of the patrolman INS, it determines the importance of the report to be higher. On the other hand, when the control terminal S1 determines that the speaker of the voice corresponding to the specific word or intention that triggered the transition to the report mode is not the voice of the patrolman INS, it determines the importance of the report to be lower.

[0071] In addition, in determining the importance of the report, the control terminal S1 may use the text interpretation result based on the text data or the emotional information of the patrolman INS based on the voice data. The control terminal S1 may determine the importance of the report (i.e., urgency) based on the likelihood that the speaker of the voice corresponding to the specific word or intention that triggered the transition to the report mode is the speech of the patrolman INS and the text interpretation result or the emotional information of the patrolman INS. Furthermore, when the control terminal S1 determines that the specific word or intention that triggered the transition to the report mode or the text interpretation result after the transition to the report mode further includes a specific word or intention such as "what are you doing," it may omit the calculation process of the importance of the report and the report waiting time and execute the report to the report destination.

[0072] This allows the control terminal S1 to determine the importance or urgency of the report with a higher degree of accuracy. In addition, this allows the control terminal S1 to accept voice input operations from someone other than the patrolman INS in situations where it is not necessary to prevent impersonation by others, such as when the patrolman INS suddenly collapses during patrol work and a person other than the patrolman INS discovers the abnormal state of the patrolman INS and reports it.

[0073] The initial state (step St100) shown in Fig. 4 will be described with reference to Fig. 5. Fig. 5 is a flowchart illustrating an example of an operation procedure in the initial state of the control terminal S1 in the first embodiment.

[0074] In the initial state, after an application for supporting the patrol work by the patrolman INS is started, the terminal device P1 accepts a login operation by the patrolman INS who performs the patrol work. The terminal device P1 transmits login information by the patrolman INS to the control terminal S1. Note that the login operation here may be performed by inputting a passcode or a keyword, or may be performed by biometric authentication using a face, iris, fingerprint, or the like.

[0075] The control terminal S1 receives the login information transmitted from the terminal device P1, and determines whether or not the patrol person INS has logged in to the application (St111).

[0076] When the control terminal S1 determines in the process of step St111 that the patrol person INS has logged in to the application (St111, YES), it identifies the patrol person INS based on the login information. The control terminal S1 holds user information of the patrol person INS who is the user using the application, that is, information about the patrol person INS (St112). In addition, the control terminal S1 extracts information about the patrol work assigned to this patrol person INS and transmits it to the terminal device P1 (St112).

[0077] The information regarding patrol work referred to here is information that supports the patrol worker INS performing the patrol work, and includes, for example, information regarding the patrol route, information regarding the patrol locations, information regarding the patrol route of each patrol location, historical information regarding patrol work performed in the past, etc.

[0078] On the other hand, when it is determined in the process of step St111 that the patrol person INS has not logged in to the application (St111, NO), the control terminal S1 returns to the process of step St111 again.

[0079] The control terminal S1 waits until the patrol worker INS starts patrol work (St113). The terminal device P1 accepts an operation to start patrol work by the patrol worker INS, generates a control command to notify the start of patrol work, and transmits it to the control terminal S1. When the control terminal S1 determines that the patrol work by the patrol worker INS has started based on the control command to notify the start of patrol work transmitted from the terminal device P1, it starts recording the voice data transmitted from the terminal device P1 and transitions from the initial state to patrol mode (St114).

[0080] The control terminal S1 accepts a voice input operation based on the voice data of the patrolman INS transmitted from the terminal device P1 (St115). The control terminal S1 determines whether or not there is a voice input operation by the patrolman INS in the patrol mode based on the voice data of the patrolman INS transmitted from the terminal device P1 (St116).

[0081] When it is determined in the process of step St116 that a voice input operation has been performed by the patrol person INS (St116, YES), the control terminal S1 starts patrol work support in the patrol mode.

[0082] On the other hand, when it is determined in the process of step St116 that the patrol person INS has not performed a voice input operation (St116, NO), the control terminal S1 returns to the process of step St116 again.

[0083] Next, the patrol mode (step St200) shown in Fig. 4 will be described with reference to Fig. 6 and Fig. 7, respectively. Fig. 6 is a flowchart for explaining an example of an operation procedure of the control terminal S1 in the patrol mode in the first embodiment. Fig. 7 is a flowchart for explaining an example of a procedure for shifting from the patrol mode to another mode in the control terminal S1 in the first embodiment. It goes without saying that the example of the operation procedure of the patrol mode shown in Fig. 6 may be repeatedly executed until a shift to another mode is made or until the patrol work by the patrol person INS is completed.

[0084] After starting the patrol work support in the patrol mode, the control terminal S1 resets a prompt counter c, which is the number of times that the patrol person INS is requested to input voice regarding the presence or absence of abnormalities in the patrol locations, to "c=0 (zero)" (St211). The control terminal S1 determines whether or not it is waiting for voice input based on the voice data transmitted from the terminal device P1 (St212).

[0085] The state of waiting for voice input here refers to a state in which, in acquiring voice data transmitted from the terminal device P1, the input of voice data containing the speech of the patrolman INS or any person other than the patrolman INS is awaited. When the control terminal S1 determines that the voice data transmitted from the terminal device P1 contains the speech of the patrolman INS or any person other than the patrolman INS, it determines that voice has been input. When the control terminal S1 determines that the voice data transmitted from the terminal device P1 does not contain the speech of the patrolman INS or any person other than the patrolman INS, it continues the state of waiting for voice input.

[0086] Furthermore, when the terminal device P1 is capable of performing noise removal processing or the patrol location is in a quiet environment and voice data not including environmental sounds or noises around the patrolman INS can be acquired from the terminal device P1, the control terminal S1 may determine whether the voice input is waiting state based on whether the voice data includes any voice such as the voice of the patrolman INS, the voice of any person other than the patrolman INS, or environmental sounds. In such a case, when the control terminal S1 determines that the voice data transmitted from the terminal device P1 includes any voice, it determines that voice has been input. When the control terminal S1 determines that the voice data transmitted from the terminal device P1 does not include any voice, it continues the voice input waiting state. Here, the control terminal S1 may perform noise removal processing on the voice data transmitted from the control terminal S1 and determine whether the voice input is waiting state based on the voice data after noise removal.

[0087] If the control terminal S1 determines in the processing of step St212 that it is not waiting for voice input (St212, NO), it performs voice recognition on the voice data transmitted from the terminal device P1 and converts the voice data into text data based on the voice recognition result (St213).

[0088] The control terminal S1 executes speaker verification or speaker identification based on the voice data transmitted from the terminal device P1 and the voice data used to authenticate the patrolman INS at the time of login (St214). As a result, the control terminal S1 determines whether the voice included in the voice data transmitted from the terminal device P1 is the voice of the patrolman INS who is logged in to the application.

[0089] The control terminal S1 executes sentence interpretation using the converted text data (St215). The control terminal S1 determines whether or not a specific word or a specific intention is included in the sentence interpretation result (St216).

[0090] The process of step St216 will be specifically described below. The process of step St216 is equivalent to the processes of steps St216A to St216E described below. Note that in the example of the operation procedure shown in FIG. 7, the processes of steps St217 and St219 are omitted for ease of understanding.

[0091] Specifically, the control terminal S1 judges whether the text interpretation result includes a specific word or a word of intent that indicates the patrol result, such as "no abnormality" or "measures taken against abnormality" (St216A). When the control terminal S1 judges in the process of step St216A that the text interpretation result includes a specific word or a word of intent that indicates the patrol result (St216A, YES), it executes an action of recording the current patrol location (location information) and the patrol work result corresponding to this text interpretation result in association with each other (St218A). In addition, the control terminal S1 executes an action of generating a UI screen or voice data that indicates the next patrol work location (location) or the patrol work content, etc., and transmitting it to the terminal device P1 (St218A).

[0092] On the other hand, when the control terminal S1 determines in the processing of step St216A that the sentence interpretation result does not contain specific words or words of intent indicating the patrol result (St216A, NO), it further determines whether or not the sentence interpretation result contains specific words or words of intent requesting a report, such as "abnormality detected," "emergency situation," "report," or "please call" (St216B).

[0093] When the control terminal S1 determines in the process of step St216B that the text interpretation result contains a specific word or a word of intent that indicates a request for a report (St216B, YES), it records the text interpretation result and transitions to a report mode (St218B). The control terminal S1 determines the importance of the report and the report time corresponding to the importance of the report, and based on the determined report time, makes a call possible between the terminal device P1 and a preset report destination (for example, the police or the manager of the monitoring center).

[0094] On the other hand, when the control terminal S1 determines in the process of step St216B that the text interpretation result does not include a specific word or word of intent indicating a request for a report (St216B, NO), it further determines whether or not the text interpretation result includes a specific word or word of intent indicating a request for interruption or extension of the patrol work, such as "interrupt" or "extension" (St216C). Note that here, the control terminal S1 may further determine whether or not the current prompt counter c is c≧1, and may determine that the text interpretation result includes a word or intent indicating an extension of the patrol work only when it is determined that the prompt counter c is c≧1.

[0095] If the control terminal S1 determines in the processing of step St216C that the sentence interpretation result contains specific words or words of intent indicating a request to interrupt or extend the patrol work (St216C, YES), it temporarily stops recording the voice data transmitted from the terminal device P1 and transitions to interrupt mode (step St300).

[0096] On the other hand, if the control terminal S1 determines in the processing of step St216C that the sentence interpretation result does not contain specific words or words of intent indicating a request to interrupt or extend the patrol work (St216C, NO), it further determines whether the sentence interpretation result contains specific words or words of intent indicating a request to wait for assistance with the patrol work, such as ``standby'' (St216D).

[0097] If the control terminal S1 determines in the processing of step St216D that the sentence interpretation result contains specific words or words of intent indicating a request to wait for assistance with patrol work (St216D, YES), it transitions to standby mode (step St400) while continuing to record the voice data transmitted from the terminal device P1.

[0098] On the other hand, if the control terminal S1 determines in the processing of step St216D that the sentence interpretation result does not contain specific words or words of intent indicating a request to wait for patrol work support (St216D, NO), it further determines whether the state in which the sentence interpretation result is a word that cannot be interpreted has continued for a certain period of time (e.g., 30 seconds, 1 minute, etc.) (St216E).

[0099] If the control terminal S1 determines in the processing of step St216E that the state in which the sentence interpretation result is an uninterpretable word has continued for a certain period of time (St216E, YES), it executes an action to record the sentence interpretation result (St218C) and transitions to a reporting mode.

[0100] When the control terminal S1 determines in the process of step St216E that the state in which the sentence interpretation result is an uninterpretable word has not continued for a certain period of time (St216E, NO), the control terminal S1 proceeds to the process of step St218.

[0101] If the control terminal S1 determines in the processing of step St216 (specifically, each of steps St216A to St216E) that the sentence interpretation result contains specific words or a specific intention (St216, YES), it performs voiceprint authentication based on the voice data transmitted from the terminal device P1 and the voiceprint data of at least one patrol officer INS registered in the voiceprint registration database DB (St217).

[0102] On the other hand, if the control terminal S1 determines in the processing of step St216 (specifically, each of steps St216A to St216E) that the sentence interpretation result does not contain a specific word or a specific intention (St216, NO), it executes an action corresponding to the sentence interpretation result (St218).

[0103] For example, when the control terminal S1 determines that the text interpretation result contains specific words or specific intentions such as "abnormal" or "report" that do not indicate the result of the patrol work, it transitions from patrol mode to report mode and executes the action of reporting to a pre-set report destination.

[0104] Based on the speaker matching or speaker identification result in step St214 or the voiceprint authentication result in step St217, the control terminal S1 determines whether or not the person who uttered the specific words or the voice with the specific intention is the same person as the patrol officer INS who logged in to the application (St219). Note that the process of step St217 is not essential and may be omitted.

[0105] If the control terminal S1 determines in the processing of step St219 that the person who spoke the specific words or the voice with a specific intent is the same person as the patrol officer INS who logged in to the application (St219, YES), it executes an action corresponding to the sentence interpretation result (St218).

[0106] For example, when the control terminal S1 determines that the sentence interpretation result includes a specific word or a specific intention such as "no abnormality", "abnormality", or "abnormality countermeasure taken" indicating the abnormal state has been resolved (St216A, YES), it determines whether the speaker of this word is the patrol worker INS (St219). When the control terminal S1 determines that the speaker is the patrol worker INS (St219, YES), it accepts and records the voice input operation of the patrol work result such as "no abnormality", "abnormality", or "abnormality countermeasure taken" indicating the abnormal state has been resolved (St218). On the other hand, when the control terminal S1 determines that the speaker is not the patrol worker INS (St219, NO), it omits the acceptance of the voice input operation of the patrol work result such as "no abnormality", "abnormality", or "abnormality countermeasure taken" indicating the abnormal state has been resolved (St220).

[0107] Also, for example, when the control terminal S1 determines that the sentence interpretation result includes a specific word or a specific intention other than the patrol work result, such as "there is an abnormality" or "report" (St216B, YES), it determines whether the speaker of this word is the patrol person INS (St219). When the control terminal S1 determines that the speaker is the patrol person INS (St219, YES), it accepts the voice input operation of "there is an abnormality" or "report", transitions from the patrol mode to the report mode, and executes the action of reporting to a preset report destination (St218). On the other hand, when the control terminal S1 determines that the speaker is not the patrol person INS (St219, NO), it omits the acceptance of the voice input operation of "there is an abnormality" or "report" (St220), and returns to step St221.

[0108] Also, for example, when the control terminal S1 determines that the sentence interpretation result contains a specific word or a specific intention such as "interrupt" or "extend" regarding the patrol work (St216C, YES), it determines whether the speaker of this word is the patrol person INS (St219). When the control terminal S1 determines that the speaker is the patrol person INS (St219, YES), it accepts a voice input operation such as "interrupt" or "extend" and executes an action to switch from the patrol mode to the interruption mode (St218). On the other hand, when the control terminal S1 determines that the speaker is not the patrol person INS (St219, NO), it omits the acceptance of a voice input operation such as "interrupt" or "extend" (St220) and returns to step St221.

[0109] Also, for example, when the control terminal S1 determines that the sentence interpretation result includes a specific word such as "standby" or a specific intention regarding the patrol work (St216D, YES), it determines whether the speaker of this word is the patrol person INS (St219). When the control terminal S1 determines that the speaker is the patrol person INS (St219, YES), it accepts a voice input operation such as "standby" and executes an action to switch from the patrol mode to the standby mode (St218). On the other hand, when the control terminal S1 determines that the speaker is not the patrol person INS (St219, NO), it omits the acceptance of a voice input operation such as "standby" (St220) and returns to step St221.

[0110] Also, for example, when the control terminal S1 determines that the sentence interpretation result is not related to the patrol work, that is, the uninterpretable word or the word of intent has continued for a certain period (first elapsed time) or more (St216E, YES), it determines whether the speaker of the word is the patrol person INS (St219). When the control terminal S1 determines that the speaker is not the patrol person INS (St219, YES), it executes an action to shift from the patrol mode to the report mode (St218). On the other hand, when the control terminal S1 determines that the speaker is the patrol person INS (St219, NO), it omits the acceptance of the voice input operation based on the voice uttered by the patrol person INS (St220) and returns to step St221.

[0111] When the control terminal S1 determines in the process of step St219 that the speaker who uttered the specific words or the specific intention is not the same person as the patrol officer INS who logged in to the application (St219, NO), it omits the execution of the action corresponding to the sentence interpretation result (St220), generates voice data or a UI screen instructing re-input of the voice, and transmits it to the terminal device P1. The terminal device P1 notifies the terminal device S1 that the action has been omitted by outputting voice based on the voice data transmitted from the control terminal S1 to the voice output device DV2 or displaying a UI screen on the display unit 13 (St221).

[0112] In addition, the control terminal S1 may count the number of times that it determines that a person who has spoken a specific word or a statement with a specific intent is not the same person as the patrol officer INS who is logged in to the application, and if it determines that the counted number of times is equal to or greater than a predetermined number (e.g., five times), it may report the report to the report destination.

[0113] On the other hand, if the control terminal S1 determines in the processing of step St212 that it is in a state waiting for voice input (St212, YES), it measures the time during which it is in the state waiting for voice input (an example of a second elapsed time) and further determines whether or not it has remained in the state waiting for voice input for a certain period of time T1 (an example of a second predetermined period of time) or more (St222).

[0114] When the control terminal S1 determines in the process of step St222 that a certain period of time T1 or more has elapsed while waiting for voice input (St222, YES), the control terminal S1 generates a prompt announcement (voice data) by voice synthesis to prompt the patrol officer INS to input voice. The control terminal S1 transmits the generated voice synthesis data of the prompt announcement to the terminal device P1. The terminal device P1 outputs the voice synthesis data transmitted from the control terminal S1 to the voice output device DV2, and issues the prompt announcement to the patrol officer INS (St223).

[0115] The reminder announcement referred to here is, for example, a voice message saying, "It will take some time before we move to the next patrol location. If you are still patrolling, say "still patrolling." If you need to be quiet for a while, say "silent mode." If there is an abnormality, please report it to the management center immediately." This is a notification that prompts voice input to allow the patrol officer INS to confirm whether or not the officer is still on patrol.

[0116] On the other hand, when the control terminal S1 determines in the process of step St222 that the certain time T1 or more has not elapsed in the voice input waiting state (St222, NO), the control terminal S1 returns to the process of step St212.

[0117] The control terminal S1 increments (c=c+1) the current prompt counter c at the time when the prompt announcement is executed (St224), and determines whether the incremented prompt counter c is equal to or greater than a threshold value CTH (an example of a second predetermined number of times) (St225).

[0118] When the control terminal S1 determines in the process of step St225 that the incremented prompt counter c is equal to or greater than the threshold CTH (St225, YES), it transitions from the patrol mode to the report mode, and generates and transmits current status information to the report destination (St226). Note that the status information here includes any one of information such as the location information of the patrol person INS, progress information of the patrol work, and the elapsed time from the timing of the last voice input by the patrol person INS (i.e., acquisition of voice data).

[0119] On the other hand, when the control terminal S1 determines in the process of step St225 that the incremented prompt counter c is not equal to or greater than the threshold value CTH (St225, NO), the control terminal S1 returns to the process of step St212.

[0120] In the process of step St216E, the control terminal S1 calculates the amount of speech of the words that are not related to the patrol work, that is, the words that cannot be interpreted or the words of intent, based on the sentence interpretation result, and when it is determined that the calculated amount of speech is equal to or greater than a predetermined amount of speech (St216E, YES), it determines whether or not the person who is speaking the predetermined amount of speech or more is the patrol person INS (St219). When the control terminal S1 determines that the person who is speaking the predetermined amount of speech or more is not the patrol person INS (St219, YES), it executes an action to shift from the patrol mode to the report mode (St218). On the other hand, when the control terminal S1 determines that the person who is speaking the predetermined amount of speech or more is the patrol person INS (St219, NO), it omits the acceptance of the voice input operation based on the speech of the patrol person INS (St220) and returns to step St221. The amount of speech may be calculated for each speaker by speaker identification.

[0121] As described above, the control terminal S1 records the patrol work results based on the voice data transmitted from the terminal device P1, and switches from the patrol mode to another mode, such as the interruption mode, the standby mode, or the notification mode, based on the sentence interpretation result and whether or not the voice corresponding to the sentence interpretation result is the spoken voice of the patrol person INS himself / herself. This allows the control terminal S1 to more effectively prevent others from impersonating the patrol person INS, and to support the patrol work based on the voice input operation of the patrol person INS.

[0122] Next, the interruption mode (step St300) shown in Fig. 4 will be described with reference to Fig. 8. Fig. 8 is a flowchart illustrating an example of an operation procedure of the control terminal S1 in the interruption mode in the first embodiment.

[0123] After the control terminal S1 has entered the interruption mode, it temporarily stops recording the voice data transmitted from the terminal device P1, and interrupts the urging announcement process requesting voice input.

[0124] The control terminal S1 measures the time that has elapsed since the transition to the interruption mode from the timing of the transition to the interruption mode. The control terminal S1 determines whether or not the time that has elapsed since the transition to the interruption mode is equal to or longer than a certain time T2 (St311). Note that the certain time T2 is greater than the certain time T1.

[0125] When the control terminal S1 determines in the process of step St311 that the elapsed time is equal to or longer than the certain time T2 (St311, YES), the control terminal S1 transitions to a notification mode and notifies a preset notification destination (St312).

[0126] On the other hand, if the control terminal S1 determines in the processing of step St311 that the elapsed time has not exceeded the certain time T2 (St311, NO), it determines whether voice data including a person's speaking voice has been transmitted from the terminal device P1 (i.e., whether there has been voice input) (St313).

[0127] If the control terminal S1 determines in the processing of step St313 that voice data including a person's speaking voice has been transmitted from the terminal device P1 (St313, YES), it performs voice recognition on the voice data and converts the voice into text data based on the voice recognition result (St314).

[0128] On the other hand, when the control terminal S1 determines in the process of step St313 that the voice data including the speech of a person is not transmitted from the terminal device P1 (St313, NO), the control terminal S1 returns to the process of step St311.

[0129] The control terminal S1 executes text interpretation based on the converted text data (St315). The control terminal S1 judges whether the text interpretation result includes a specific word or a specific intention for transitioning from the interruption mode to the cycling mode (St316). The control terminal S1 judges whether the text interpretation result includes a specific word or a word of intention, such as "return to cycling mode", indicating a request to end the interruption mode, or to transition or return from the interruption mode to the cycling mode.

[0130] If the control terminal S1 determines in the processing of step St316 that the voice data transmitted from the terminal device P1 contains specific words or a specific intention for transitioning from the interruption mode to the loop mode (St316, YES), the control terminal S1 resumes recording of the voice data transmitted from the terminal device P1 and the prompt announcement processing requesting voice input, and transitions to the loop mode (step St200).

[0131] On the other hand, when the control terminal S1 determines in the process of step St316 that a specific word or a specific intention for switching from the interruption mode to the loop mode is not included (St316, NO), the control terminal S1 returns to the process of step St311.

[0132] As described above, the control terminal S1 in the interruption mode can accept only voice input operations requesting the end of the interruption mode, or the transition from the interruption mode to the loop mode or the return thereto, based on the voice data transmitted from the terminal device P1.

[0133] Next, the standby mode (step St400) shown in Fig. 4 will be described with reference to Fig. 9. Fig. 9 is a flowchart illustrating an example of an operation procedure of the control terminal S1 in the standby mode in the first embodiment.

[0134] After the control terminal S1 has entered the standby mode, it interrupts the urging announcement process for requesting voice input while continuing to record the voice data transmitted from the terminal device P1.

[0135] The control terminal S1 determines whether or not voice data including a person's speech has been transmitted from the terminal device P1 (that is, whether or not voice input has been made) (St411).

[0136] If the control terminal S1 determines in the processing of step St411 that voice data including a person's speaking voice has been transmitted from the terminal device P1 (St411, YES), it performs voice recognition on the voice data and converts the voice into text data based on the voice recognition result (St412).

[0137] On the other hand, when the control terminal S1 determines in the process of step St411 that the voice data including the speech of a person is not transmitted from the terminal device P1 (St411, NO), the control terminal S1 returns to the process of step St411.

[0138] The control terminal S1 executes sentence interpretation based on the converted text data (St413). The control terminal S1 determines whether or not the sentence interpretation result includes a specific word or a specific intention for switching from the standby mode to the cycling mode (St414).

[0139] When the control terminal S1 determines in the process of step St414 that the voice input includes a specific word or a specific intention for switching from the standby mode to the patrol mode (St414, YES), the control terminal S1 resumes the prompt announcement process requesting voice input and switches to the patrol mode (step St200). When the control terminal S1 determines that the sentence interpretation result includes a specific word or a word of intention, such as "return to patrol mode," indicating the end of the standby mode, or a request for switching from the standby mode to the patrol mode or returning, the control terminal S1 switches to the patrol mode (step St200).

[0140] Furthermore, when the control terminal S1 determines in the process of step St414 that a specific word or a specific intention for switching from the interruption mode to the patrol mode is included (St414, OTHER), the process returns to the process of step St411. When the control terminal S1 determines that the sentence interpretation result includes a specific word or a word of intention indicating danger or urgency, such as "What are you doing" or "Dangerous", the control terminal S1 switches to a report mode and reports to a preset report destination (St416).

[0141] Moreover, when the control terminal S1 determines in the process of step St414 that a specific word or a specific intention for switching from the interruption mode to the loop mode is not included (St414, NO), the control terminal S1 returns to the process of step St411.

[0142] As described above, the control terminal S1 in the standby mode only accepts voice input operations requesting the end of the standby mode, or the transition or return from the standby mode to the patrol mode, based on the voice data transmitted from the terminal device P1. This makes it possible for the control terminal S1 to prevent erroneous patrol work support or erroneous reporting based on the content of speech of the patrol person INS or a person other than the patrol person INS when the patrol person INS is performing work other than patrol work (for example, talking to a third party, tracking a suspicious person, etc.).

[0143] (Embodiment 2) The voice input system 100 according to the first embodiment has been described as an example in which the terminal device P1 realizes a UI display function, and the control terminal S1 realizes functions such as voice recognition, voiceprint authentication, voiceprint registration, behavior recording, feature extraction, behavior recording, etc. The voice input system 100A according to the second embodiment will be described as an example in which the terminal device P1 realizes functions such as UI display, voice recognition, voiceprint authentication, voiceprint registration, behavior recording, feature extraction, etc., and the control terminal S1 realizes a behavior recording function.

[0144] A description will be given of a voice input system 100A according to the second embodiment. Fig. 10 is a diagram showing an example of the overall configuration of the voice input system according to the second embodiment.

[0145] The voice input system 100A according to the second embodiment includes a terminal device P1A, a network NW, a control terminal S1A, and a monitor MN. The voice input system 100A may not include the monitor MN. The voice input system 100A may include a plurality of terminal devices and a plurality of control terminals.

[0146] The terminal device P1A is a terminal device carried by a patrol worker INS during the patrol work. The terminal device P1A is connected to the control terminal S1A via the network NW so that data can be transmitted and received. The terminal device P1A realizes functions such as UI display, voice recognition, voiceprint authentication, voiceprint registration, behavior recording, feature extraction, etc. by the processor 11.

[0147] Specifically, the processor 11 in the terminal device P1A is configured to be able to realize the functions of the voice recognition unit 211, the sentence interpretation unit 212, the feature extraction unit 213, the voiceprint processing unit 214, the action determination unit 215, the voice synthesis unit 216, and the report communication unit 217 of the control terminal S1, in addition to the configuration of the terminal device P1 shown in Fig. 2. Furthermore, the terminal device P1A is provided with a voiceprint registration database DB in which the voiceprint data of at least one patrol officer INS who can operate the terminal device P1A can be registered.

[0148] The control terminal S1A is realized by a PC, a notebook PC, a tablet terminal, a smartphone, a cloud server, or the like installed in a monitoring center that monitors or manages the patrol work performed by at least one patrol person INS. The control terminal S1A realizes an action recording function that records information on the patrol results transmitted from the terminal device P1A by the processor 21.

[0149] Specifically, the control terminal S1A is configured such that the configuration of the processor 21 of the terminal device P1 shown in FIG.

[0150] Here, the correspondence between the processes performed by the terminal device P1A according to the second embodiment and the processes shown in the flowcharts of FIGS. 5 to 9 will be described.

[0151] The terminal device P1A executes the processes of steps St111 to St116 in the operation procedure (step St100) in the initial state shown in FIG.

[0152] The terminal device P1A executes processes other than steps St218A, St218B, and St218C in the operation procedure (step St200) of the cyclic mode shown in each of Fig. 6 and Fig. 7. Note that steps St218A, St218B, and St218C are executed by the control terminal S1A as in the first embodiment.

[0153] The terminal device P1A executes the processes of steps St311 to St317 in the operation procedure (step St300) in the interruption mode shown in FIG.

[0154] The terminal device P1A executes the processes of steps St411 to St415 in the operation procedure (step St400) in the standby mode shown in FIG.

[0155] (Additional Note) The above description of each embodiment discloses the following techniques.

[0156] (Technology 1) An acquisition unit (communication unit 20) that acquires voice data of a worker performing a task; a sentence interpretation unit 212 that performs speech recognition on the voice data and performs sentence interpretation on the recognized speech content; a determination unit (voiceprint authentication unit 214B) that determines whether or not a speech corresponding to the sentence interpretation result of the voice data is a speech of the worker based on the sentence interpretation result of the speech content; and a control unit (action determination unit 215) that accepts an input operation corresponding to the sentence interpretation result based on the sentence interpretation result and whether or not the spoken voice corresponding to the sentence interpretation result is the spoken voice of the operator. Voice input device (control terminal S1). With this configuration, the control terminal S1 can determine whether to accept a voice input operation of the task result (e.g., "abnormality detected," "no abnormality detected," etc.) included in the text interpretation result, based on the text interpretation result and whether the voice corresponding to the text interpretation result is the spoken voice of the patrolman INS himself. This allows the control terminal S1 to more effectively suppress spoofing by the patrolman INS and execute the acceptance of the voice input operation of the patrolman INS.

[0157] (Technology 2) When the judgment unit (voiceprint authentication unit 214B) judges that the sentence interpretation result indicates that the work is normal, it judges whether the voice uttered indicating that the work is normal is a voice uttered by the worker, When the control unit (action determination unit 215) determines that the voice indicating that the work is normal is a voice uttered by the worker, it accepts and records an input operation indicating that the work is normal. A voice input device as described in (Technology 1). With this configuration, the control terminal S1 can accept the voice input operation only when it determines that the sentence interpretation result includes the patrol work result "no abnormality" as a specific word or specific intention, and also determines that the spoken voice corresponding to the word or intention of the patrol work result "no abnormality" is the spoken voice of the patrol officer INS himself. Furthermore, the control terminal S1 performs speaker matching or speaker identification (i.e., identity authentication) only when it determines that the sentence interpretation result includes the patrol work result "no abnormality" as a specific word or specific intention, and can therefore perform speaker matching or speaker identification (i.e., identity authentication) only for the spoken voice corresponding to the voice input operation, even if the voice collected by the terminal device P1 includes a voice other than that of the patrol officer INS. This allows the control terminal S1 to more effectively suppress impersonation by the patrol officer INS and accept the voice input operation of the patrol officer INS.

[0158] (Technology 3) When the judgment unit (voiceprint authentication unit 214B) judges that the sentence interpretation result indicates that the work is abnormal, it omits judging whether or not the voice uttered indicating that the work is abnormal is a voice uttered by the worker, The control unit (action determination unit 215) receives and records an input operation indicating that the work is abnormal. A voice input device according to (Technology 1) or (Technology 2). With this configuration, when the control terminal S1 determines that the sentence interpretation result includes the patrol work result "abnormal" as a specific word or specific intention, and determines that the spoken voice corresponding to the word or intention of the patrol work result "abnormal" is not the spoken voice of the patrol officer INS, it can omit accepting the voice input operation. Furthermore, the control terminal S1 performs speaker matching or speaker identification (i.e., identity authentication) only when it determines that the sentence interpretation result includes the patrol work result "abnormal" as a specific word or specific intention, and can perform speaker matching or speaker identification (i.e., identity authentication) only for the spoken voice corresponding to the voice input operation, even if the voice collected by the terminal device P1 includes a voice other than that of the patrol officer INS. This allows the control terminal S1 to more effectively suppress impersonation by the patrol officer INS and accept the voice input operation of the patrol officer INS.

[0159] (Technology 4) When the determination unit (voiceprint authentication unit 214B) determines that the sentence interpretation result indicates that the abnormality has been resolved, it determines whether the voice uttered indicating that the abnormality has been resolved is a voice uttered by the operator, When the control unit (action determination unit 215) determines that the voice uttered indicating that the abnormality has been resolved is not the voice uttered by the worker, the control unit (action determination unit 215) rejects the input operation indicating that the abnormality has been resolved. The voice input device according to any one of (Technology 1) to (Technology 3). With this configuration, the control terminal S1 can omit accepting the voice input operation only when it determines that the sentence interpretation result includes the patrol work result "resolve the abnormality" as a specific word or specific intention, and also determines that the spoken voice corresponding to the word or intention of the patrol work result "resolve the abnormality" is not the spoken voice of the patrolman INS himself. Furthermore, the control terminal S1 performs speaker matching or speaker identification (i.e., identity authentication) only when it determines that the sentence interpretation result includes the patrol work result "abnormality exists" as a specific word or specific intention, and can perform speaker matching or speaker identification (i.e., identity authentication) only for the spoken voice corresponding to the voice input operation, even if the voice collected by the terminal device P1 includes a voice other than that of the patrolman INS. This allows the control terminal S1 to more effectively suppress impersonation by the patrolman INS and accept the voice input operation of the patrolman INS.

[0160] (Technology 5) When the judgment unit (voiceprint authentication unit 214B) judges that the sentence interpretation result indicates that the work is abnormal, it judges whether the voice uttered indicating that the work is abnormal is a voice uttered by the worker, When the control unit (action determination unit 215) determines that the voice indicating that the work is abnormal is a voice of the worker, it accepts an input operation indicating that the work is abnormal and notifies an external terminal of the abnormality. The voice input device according to any one of (Technology 1) to (Technology 4). With this configuration, the control terminal S1 can accept the voice input operation only when it determines that the sentence interpretation result includes the patrol work result "abnormal" as a specific word or specific intention, and also determines that the spoken voice corresponding to the word or intention of the patrol work result "abnormal" is the spoken voice of the patrol officer INS himself. Furthermore, the control terminal S1 performs speaker matching or speaker identification (i.e., identity authentication) only when it determines that the sentence interpretation result includes the patrol work result "abnormal" as a specific word or specific intention, and can therefore perform speaker matching or speaker identification (i.e., identity authentication) only for the spoken voice corresponding to the voice input operation, even if the voice collected by the terminal device P1 includes a voice other than that of the patrol officer INS. This allows the control terminal S1 to more effectively suppress impersonation by the patrol officer INS and accept the voice input operation of the patrol officer INS.

[0161] (Technology 6) The determination unit (voiceprint authentication unit 214B) measures a first elapsed time during which the voice data does not include any utterance content related to the work, based on the sentence interpretation result, and when the first elapsed time is equal to or longer than a first predetermined time, determines whether or not the utterance voice of the voice data is the utterance voice of the worker; The control unit (action determination unit 215) determines the timing of reporting to a pre-set reporting destination based on whether the spoken voice of the voice data is the spoken voice of the worker, and reports the determined timing. The voice input device according to any one of (Technology 1) to (Technology 5). With this configuration, the control terminal S1 can determine whether to report to the report destination based on the presence or absence of the speech of the patrolman INS included in the voice data when the time when no voice input operation is performed continues for more than the first predetermined time based on the sentence interpretation result. This allows the control terminal S1 to check the condition of the patrolman INS more quickly even if something happens to the patrolman INS during patrol work by reporting to the report destination. Note that the first elapsed time here may be a time obtained by multiplying the certain time T1 measured in the process of step St222 by the upper limit value of the reminder counter cn (i.e., the value of the threshold CTH), or may be a certain time T2 measured in the process of step St311.

[0162] (Technology 7) The determination unit (voiceprint authentication unit 214B) calculates the amount of speech included in the sentence interpretation result, and when it is determined that the amount of speech is equal to or greater than a predetermined amount, determines whether or not the speech voice corresponding to the amount of speech is the speech voice of the worker; The control unit (action determination unit 215) determines the timing of reporting to a pre-set reporting destination based on whether the speech voice corresponding to the speech volume is the speech voice of the worker, and reports the determined timing. The voice input device according to any one of (Technology 1) to (Technology 6). With this configuration, when the control terminal S1 determines that the amount of speech of the patrolman INS or a person other than the patrolman INS included in the text interpretation result is equal to or greater than a predetermined amount, it can determine whether or not to report to the report destination based on whether or not this speech is the speech of the patrolman INS. As a result, the control terminal S1 can check the status of the patrolman INS more quickly by determining whether or not someone other than the patrolman INS is involved in trouble and reporting it to the report destination.

[0163] (Technology 8) The determination unit (voiceprint authentication unit 214B) measures a second elapsed time during which the voice data does not include the speech content based on the sentence interpretation result, and determines whether or not the second elapsed time is equal to or longer than a second predetermined time; The control unit (action determination unit 215) generates and outputs a prompting notice to prompt the worker to speak when the determination unit (voiceprint authentication unit 214B) determines that the second elapsed time is equal to or greater than the second predetermined time. The voice input device according to any one of (Technology 1) to (Technology 7). With this configuration, when the control terminal S1 determines that the voice data does not include the speech of any person for a period of time equal to or longer than the second elapsed time (certain time T1), it requests the patrol officer INS to urge him / her to speak. This allows the control terminal S1 to determine whether the patrol officer INS is working or not, whether the patrol work is continuing or not, etc.

[0164] (Technology 9) The control unit (action determination unit 215) counts the number of times a prompt notification is generated to prompt the worker to speak, and when it is determined that the number of times the prompt notification is generated is equal to or greater than a second predetermined number, issues a report to a previously set report destination. A voice input device according to (Technical 8). With this configuration, when the control terminal S1 determines that the number of times the patrolman INS is prompted to speak is equal to or greater than a second predetermined number (threshold CTH), the control terminal S1 requests the patrolman INS to speak. This allows the control terminal S1 to, for example, determine whether or not someone other than the patrolman INS is involved in trouble and report the result to the reporting destination, thereby enabling the control terminal S1 to check the status of the patrolman INS more quickly.

[0165] (Technology 10) the determination unit (voiceprint authentication unit 214B) temporarily suspends measurement of the second elapsed time when it determines that the sentence interpretation result indicates that the work is to be temporarily suspended; A voice input device according to (Technical 8). With this configuration, when the control terminal S1 determines that the sentence interpretation result indicates that the work should be temporarily suspended as a specific word or a specific intention, it can stop urging the patrol staff INS to speak by temporarily suspending the measurement of the time during which the voice data does not include the speech of any person. This allows the control terminal S1 to more effectively suppress the acceptance of erroneous voice input operations due to speech intended for purposes other than patrol work, for example, when the patrol staff INS is performing work other than patrol work.

[0166] (Technology 11) When the judgment unit (voiceprint authentication unit 214B) judges that the sentence interpretation result indicates that the work is to be temporarily suspended, it judges whether the spoken voice indicating that the work is to be temporarily suspended is a spoken voice of the worker or not, When the control unit (action determination unit 215) determines that the voice indicating that the work is to be temporarily suspended is not a voice uttered by the worker, the control unit (action determination unit 215) continues measuring the second elapsed time. A voice input device according to (Technical 8). With this configuration, when the control terminal S1 determines that the words or intention to temporarily suspend work, which are included in the sentence interpretation result, are not spoken by the patrolman INS, it determines that the voice input operation is not by the patrolman INS, and continues measuring the time during which the voice data does not include the voice of any person. This allows the control terminal S1 to more effectively suppress spoofing by the patrolman INS and to accept the voice input operation of the patrolman INS.

[0167] (Technology 12) A terminal device P1 that picks up the voice of a worker performing a task; A voice input system 100 including a control terminal S1 capable of communicating with a terminal device, The terminal device P1 Transmitting the voice data of the worker to the terminal device P1; The control terminal S1 Acquire audio data, Speech recognition is performed on the speech data, and sentence interpretation is performed on the recognized speech content; Based on the sentence interpretation result of the speech content, it is determined whether or not the speech data corresponding to the sentence interpretation result is a speech of the worker; accepting an input operation corresponding to the sentence interpretation result based on the sentence interpretation result and whether or not the spoken voice corresponding to the sentence interpretation result is a spoken voice of the worker; Voice input system. With this configuration, the voice input system can determine whether to accept a voice input operation of the task result (e.g., "abnormality detected," "no abnormality detected," etc.) included in the sentence interpretation result based on the sentence interpretation result and whether the voice corresponding to the sentence interpretation result is the spoken voice of the patrolman INS himself / herself. This allows the voice input system 100 to more effectively suppress spoofing by the patrolman INS and to accept the voice input operation of the patrolman INS.

[0168] (Technology 13) A voice input method performed by a computer (control terminal S1), Acquire voice data of the worker performing the work, Speech recognition is performed on the speech data, and sentence interpretation is performed on the recognized speech content; Based on the sentence interpretation result of the speech content, it is determined whether or not the speech data corresponding to the sentence interpretation result is a speech of the worker; accepting an input operation corresponding to the sentence interpretation result based on the sentence interpretation result and whether or not the spoken voice corresponding to the sentence interpretation result is a spoken voice of the worker; Audio input method. With this configuration, the control terminal S1 can determine whether to accept a voice input operation of the task result (e.g., "abnormality detected," "no abnormality detected," etc.) included in the text interpretation result, based on the text interpretation result and whether the voice corresponding to the text interpretation result is the spoken voice of the patrolman INS himself. This allows the control terminal S1 to more effectively suppress spoofing by the patrolman INS and execute the acceptance of the voice input operation of the patrolman INS.

[0169] Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications, corrections, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure. In addition, the components in the various embodiments described above may be arbitrarily combined within the scope of the invention. [Industrial Applicability]

[0170] INDUSTRIAL APPLICABILITY The present disclosure is useful as a voice input device, a voice input system, and a voice input method that can more effectively prevent spoofing by others when accepting voice input. [Explanation of symbols]

[0171] 10,20 Communications Department 11,21 Processor 12,22 Memory 13 Display section 100,100A Voice Input System 211 Voice Recognition Unit 212 Text Interpretation Department 213 Feature Extraction Unit 214 Voiceprint Processing Unit 214A Voiceprint Registration Department 214B Voiceprint Recognition Unit 215 Action Decision Unit 216 Voice synthesis unit 217 Reporting and Communications Department 218 Behavioral Records Department DB Voiceprint Registration Database DV1 Audio Input Device DV2 Audio Output Device INS Patrol Officer P1, P1A terminal equipment S1, S1A control terminal

Claims

1. an acquisition unit that acquires voice data of a worker performing a task; a sentence interpretation unit that performs speech recognition on the voice data and performs sentence interpretation on the recognized speech content; a determination unit that determines whether or not a speech voice corresponding to the sentence interpretation result of the speech content in the voice data is a speech voice of the worker based on the sentence interpretation result of the speech content; and a control unit that receives an input operation corresponding to the sentence interpretation result based on the sentence interpretation result and whether or not the spoken voice corresponding to the sentence interpretation result is the spoken voice of the operator. Audio input device.

2. When the determination unit determines that the sentence interpretation result indicates that the work is normal, the determination unit determines whether or not a speech voice indicating that the work is normal is a speech voice of the worker, When the control unit determines that the voice indicating that the work is normal is the voice of the worker, the control unit accepts and records an input operation indicating that the work is normal. The voice input device according to claim 1 .

3. When the determination unit determines that the sentence interpretation result indicates that the work is abnormal, the determination unit omits determining whether or not the speech indicating that the work is abnormal is a speech of the worker, The control unit receives and records an input operation indicating that the work is abnormal. The voice input device according to claim 1 .

4. When the determination unit determines that the sentence interpretation result indicates that the abnormality has been resolved, the determination unit determines whether or not a speech voice indicating that the abnormality has been resolved is a speech voice of the operator, When the control unit determines that the voice indicating that the abnormality has been resolved is not the voice of the operator, the control unit rejects an input operation indicating that the abnormality has been resolved. The voice input device according to claim 3.

5. When the determination unit determines that the sentence interpretation result indicates that the work is abnormal, the determination unit determines whether or not a speech voice indicating that the work is abnormal is a speech voice of the worker, When the control unit determines that the voice indicating that the work is abnormal is the voice of the worker, the control unit accepts an input operation indicating that the work is abnormal and notifies an external terminal of the abnormality. The voice input device according to claim 1 .

6. the determination unit measures a first elapsed time during which the voice data does not include any utterance content related to the work, based on the sentence interpretation result, and when the first elapsed time is equal to or greater than a first predetermined time, determines whether or not the utterance voice of the voice data is the utterance voice of the worker; The control unit determines a timing of reporting to a preset reporting destination based on whether or not the spoken voice of the voice data is the spoken voice of the worker, and reports the determined timing. The voice input device according to claim 1 .

7. the determination unit calculates an amount of speech included in the sentence interpretation result, and when it is determined that the amount of speech is equal to or greater than a predetermined amount, determines whether or not a speech voice corresponding to the amount of speech is a speech voice of the worker; The control unit determines a timing of a notification to a preset notification destination based on whether or not the speech voice corresponding to the amount of speech is the speech voice of the worker, and notifies the worker. The voice input device according to claim 1 .

8. the determination unit measures a second elapsed time during which the voice data does not include any speech content based on the sentence interpretation result, and determines whether the second elapsed time is equal to or longer than a second predetermined time; The control unit generates and outputs a prompting notification that prompts the worker to speak when the determination unit determines that the second elapsed time is equal to or longer than a second predetermined time. The voice input device according to claim 1 .

9. The control unit counts the number of times a prompt notification is generated to prompt the worker to speak, and when it is determined that the number of times the prompt notification is generated is equal to or greater than a second predetermined number, issues a notification to a preset notification destination. The voice input device according to claim 8.

10. when the determination unit determines that the sentence interpretation result indicates that the task is to be temporarily suspended, the determination unit temporarily suspends measurement of the second elapsed time. The voice input device according to claim 8.

11. When the determination unit determines that the sentence interpretation result indicates that the work is to be temporarily suspended, the determination unit determines whether or not a speech indicating that the work is to be temporarily suspended is a speech of the worker; When the control unit determines that the voice indicating that the work is to be temporarily suspended is not a voice of the worker, the control unit continues measuring the second elapsed time. The voice input device according to claim 8.

12. A terminal device that picks up the voice of a worker performing a task; A voice input system comprising: a control terminal capable of communicating with the terminal device, The terminal device Transmitting voice data of the worker to the terminal device; The control terminal Acquiring the audio data; performing speech recognition on the speech data and performing sentence interpretation on the recognized speech content; determining whether or not a speech voice corresponding to the sentence interpretation result among the voice data is a speech voice of the worker based on the sentence interpretation result of the speech content; accepting an input operation corresponding to the sentence interpretation result based on the sentence interpretation result and whether or not the spoken voice corresponding to the sentence interpretation result is the spoken voice of the worker; Voice input system.

13. A speech input method performed by a computer, comprising the steps of: Acquire voice data of the worker performing the work, performing speech recognition on the speech data and performing sentence interpretation on the recognized speech content; determining whether or not a speech voice corresponding to the sentence interpretation result among the voice data is a speech voice of the worker based on the sentence interpretation result of the speech content; accepting an input operation corresponding to the sentence interpretation result based on the sentence interpretation result and whether or not the spoken voice corresponding to the sentence interpretation result is the spoken voice of the worker; Audio input method.

Images