Information processing system, voice operation system, program, and voice operation method

Abstract

To provide an information processing system capable of preventing a person who is not a participant of a conference from controlling an apparatus that may be used in the conference.SOLUTION: An information processing system 100 for controlling an apparatus 200 to be used in a conference based on audio data sent from a terminal device 4, comprises: a communication unit for receiving, from the terminal device, user identification information held by the terminal device and the audio data; and a command conversion unit for converting the audio data into a command for controlling the apparatus. The communication unit sends the command to the apparatus if information concerning participants of the conference includes the user identification information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system, a voice operation system, a program, and a voice operation method. [Background technology]

[0002] Display devices that display handwritten data, character strings converted from handwritten data, etc., on a display are known. Display devices equipped with relatively large touch panels are placed in conference rooms, public facilities, etc., and are used by multiple users as electronic whiteboards, etc.

[0003] There is known a technology for accepting device operations using text data generated by voice recognition (see, for example, Patent Document 1). Patent Document 1 discloses a technology for determining whether or not a device will respond to a user request input by voice together with user identification information. Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional technology has a problem that devices that can be used in a conference can be operated by people who are not participants in the conference. In other words, when no one is in the conference room, any user can enter the conference room, but it is possible for the user to display materials for the conference that the user does not have the authority to view.

[0005] In view of the above-mentioned problems, an object of the present invention is to provide an information processing system that can prevent persons who are not participants in a conference from operating devices that may be used in the conference. [Means for solving the problem]

[0006] In view of the above-mentioned problems, the present invention provides an information processing system for operating equipment used in a conference based on voice data transmitted from a terminal device, the information processing system including: a communication unit that receives, from the terminal device, user identification information and voice data held by the terminal device; a command conversion unit that converts the voice data into a command for operating the equipment; A determination unit that determines whether or not information about participants in a conference includes identification information of the user; having When the determination unit determines that the information about the participants of the conference includes identification information of the user, the determination unit further determines whether or not the information about the participants of the conference includes information that the user has entered the conference room, and when the user has entered the conference room, the communication unit The command is transmitted to the device. Effect of the Invention

[0007] It is possible to provide an information processing system that can prevent persons who are not participants in a conference from operating devices that may be used in the conference. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of a voice operation system. [Diagram 2] 11A and 11B are diagrams illustrating an operation of a user holding an IC card over a display device. [Diagram 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a display device. [Figure 4] FIG. 2 illustrates an example of a hardware configuration of an information processing system. [Diagram 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of an IC card. [Figure 6] FIG. 2 illustrates an example of a hardware configuration of a terminal device. [Figure 7] FIG. 2 is an example of a functional block diagram illustrating functions of a terminal device, an information processing system, and a display device, divided into blocks. [Figure 8] 11 is a diagram illustrating an example of command information stored in a command information storage unit. FIG. [Figure 9] FIG. 4 is a diagram showing an example of schedule information stored in a schedule information storage unit; [Figure 10] FIG. 1 is an example of a sequence diagram illustrating an operation of the voice operation system. [Figure 11]FIG. 2 is a diagram illustrating information transmitted from a terminal device to an information processing system. [Figure 12] 11A and 11B are diagrams illustrating details of a method for determining whether or not to permit voice operation by a determining unit. [Figure 13] FIG. 13 is a diagram showing an example of a display of materials by executing a command. [Figure 14] FIG. 11 is a diagram showing an example of schedule information (meeting information) registered in the schedule information (Example 2); [Figure 15] FIG. 11 is a diagram for explaining details of a method for determining whether or not to permit voice operation by a determining unit (Example 2). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A voice operation system and a voice operation method performed by the voice operation system will be described below with reference to the drawings as an example of an embodiment of the present invention. EXAMPLES

[0010] <Outline of device control> First, an overview of operation of the display device 200 using text data that has been voice-recognized will be described with reference to Fig. 1. Fig. 1 is an example of a system configuration diagram of a voice operation system 1. Details of the voice operation system 1 will be described later.

[0011] (i) User A, who will be participating in the conference, enters the conference room 7. For example, user A has his / her IC card read by the display device 200. The display device 200 transmits user A's entry information (user ID and entry information) to the schedule management device 6. Note that the conference room 7 does not have to be a room dedicated to the conference, and can be any space in which conference participants are present. For example, the space in which the display device 200 is located can be the conference room 7. The schedule management device 6 registers in the conference information that user A has entered the room.

[0012] (ii) Each user A owns a terminal device 4 such as a smartphone, and an application (hereinafter, simply referred to as an app) runs on the terminal device 4. This app is dedicated to the voice operation system 1, and transmits user A's voice data to the voice analysis device 5. User A's account information is registered in the app. When user A speaks to the terminal device 4, the terminal device 4 transmits user A's account information (user ID) and the voice data to the voice analysis device 5.

[0013] (iii) The voice analysis device 5 obtains conference information (such as the time of the conference, the account information of the participants, and the conference room ID) from the schedule management device 6 that manages the schedule of the conference, and judges whether or not the user A who spoke is a participant of the conference and has actually entered the conference room 7. In the explanation of FIG. 1, entry into the conference room 7 means having the display device 200 read an IC card. However, if there is an IC card reader 8 or the like at the entrance of the conference room 7 and entry is permitted by the IC card reader 8, whether or not the user has entered the conference room 7 may be judged depending on whether or not the IC card has been read by the IC card reader 8 at the entrance.

[0014] Alternatively, the terminal device 4 may transmit location information detected by an indoor GPS or a beacon device together with the voice data to the voice analysis device 5. The voice analysis device 5 identifies the conference room 7 based on information associating the location of each conference room 7 with the conference room ID, and determines whether or not user A, who has spoken in the conference room 7, is registered as a participant.

[0015] (iv) If the user A who spoke is a participant of the conference and is actually in the conference room 7, the voice analysis device 5 analyzes the voice data transmitted from the terminal device 4 and converts it into a command for controlling the display device 200. Examples of commands include controlling the power supply of the display device 200, displaying materials or the like on the display device 200, displaying text data generated by voice recognition on the display device 200, and the like.

[0016] In this way, when the speaker is a participant of the conference and is actually in the conference room 7, the voice analysis device 5 operates the display device 200 based on the voice data of the speaker. Therefore, a person registered as a participant of the conference can control the devices in the conference room 7.

[0017] <Terminology> A meeting is a gathering to discuss something. A meeting may also be called a convention, gathering, meeting, discussion, presentation, conference, discussion, talk, deliberation, etc.

[0018] A command is a signal or instruction that instructs the execution of a specific process. In this embodiment, the command is used to mean a command related to the operation of a device.

[0019] The input means may be any means capable of specifying coordinates on a touch panel and writing by hand, such as a pen, a human finger or hand, or a rod-shaped member.

[0020] A stroke refers to a series of operations in which a user presses an input means against a display, moves it continuously, and then releases it from the display. Stroke data is information displayed on the display based on a trajectory of coordinates input by the input means. Stroke data may be appropriately interpolated. Handwritten data is data having one or more stroke data. Handwritten input refers to handwritten data being input by a user.

[0021] An object that is displayed on a display based on stroke data is called an object. Object means a target, but in this embodiment, it means a display target. The character string converted from handwritten data through character recognition may include not only text data, but also data displayed based on user operations, such as fixed characters such as "Done," stamps displayed as marks, shapes such as circles and stars, and straight lines.

[0022] <System configuration> The system configuration of a voice operation system 1 will be described with reference to Fig. 1. The voice operation system 1 has a display device 200, which is a device to be voice-operated, and an information processing system 100. The voice operation system 1 also includes a terminal device 4 as necessary. The display device 200 is installed in a conference room 7 that a user enters, and the display device 200 and the information processing system 100 can communicate with each other via a network.

[0023] The display device 200 has a large display with a touch panel, detects the coordinates of the display indicated by the user using the input means, and displays strokes by connecting these coordinate points. Depending on the input mode set by the user, it can perform character recognition on handwritten data or display text data that has been voice-recognized. In addition, it can display images from a connected terminal device 4 and communicate with a display device 200 at another location to synchronize handwritten data. The display device 200 is also called an electronic whiteboard. An electronic whiteboard may also be called an electronic information board or an electronic whiteboard. The display device 200 does not necessarily have to have a large display with a touch panel, and may be a tablet terminal of about 5 to 10 inches.

[0024] In this embodiment, the device to be operated by voice is the display device 200, but the device may be any device that can be remotely controlled by communication. The device may be, for example, a projector, a multifunction printer, a microphone, a speaker, an air conditioner, a lighting device, or other devices, or may be a plurality of devices.

[0025] In addition, the device is not limited to a device shared by multiple participants in a conference, and any device that can be used in a conference can be a target of voice operation. For example, a general-purpose tablet terminal or PC can be a device.

[0026] The display device 200 has an IC card reader. When an IC card or a terminal device 4 having an IC card function is held over the display device 200, the display device 200 reads identification information (user ID) for identifying the user from the terminal device 4 and transmits it to the information processing system 100.

[0027] It should be noted that the IC card reader does not have to be integrated with the display device 200. The IC card reader may exist separately from the display device 200. The IC card reader may be located in the space where the conference is held.

[0028] Furthermore, the user may notify the information processing system 100 of his / her entry into the conference room 7 by a method other than an IC card. For example, the user may log in to the information processing system 100 by inputting a user ID and password into the display device 200 or an input device provided at the entrance of the conference room 7. Biometric authentication information such as a fingerprint or a face image may be used as the login method.

[0029] In this embodiment, the terminal device 4 may be any device capable of acquiring voice data and transmitting it to the information processing system 100. The terminal device 4 may be, for example, a smartphone, a PC (Personal Computer), a tablet terminal, a mobile phone, a PDA (Personal Digital Assistant), or other device having a microphone and a communication function.

[0030] The terminal device 4 functions as a voice input device that collects voice data uttered by a user. The terminal device 4 transmits the user's voice collected by a microphone to the information processing system 100 together with information for identifying the user.

[0031] The IC card also stores information for identifying a user. More specifically, in the information processing system 100, the identification information stored in the IC card is associated with information for identifying a user. The IC card may be any storage device having a memory in which the identification information for identifying a user is stored. For example, a USB memory for storing the user's identification information may be attached to the display device 200.

[0032] In addition, in this embodiment, the IC card and the terminal device 4 are separate entities, but the terminal device 4 may have an IC card function.

[0033] The information processing system 100 has one or more information processing devices. In this embodiment, the information processing system 100 has a schedule management device 6 and a voice analysis device 5. Note that, in this embodiment, the schedule management device 6 and the voice analysis device 5 are included in one information processing system 100, but the schedule management device 6 and the voice analysis device 5 may be arranged in separate servers.

[0034] The information processing system 100 may be realized by cloud computing. A cloud computer is a form of use in which resources on a network are used without being aware of specific hardware resources. The information processing system 100 may exist on-premise or on the Internet.

[0035] The schedule management device 6 manages schedule information for each user. One type of user's schedule information is conference information. In other words, the schedule information includes conference information. For each user ID, the schedule management device 6 registers the start time, end time, conference name, organizer, participants, location, and whether the user has entered the conference room 7. The schedule information may include information other than conference information.

[0036] The voice analysis device 5 receives the user's voice data from the terminal device 4. The voice data is accompanied by the speaker's user ID and information on the time of speech. The voice analysis device 5 queries the schedule management device 6 for schedule information of the user with the received user ID, and determines whether the user is a conference participant and has already entered the conference room 7. If the user is a conference participant and has already entered the conference room 7, the voice analysis device 5 analyzes the voice data included in the voice information and converts it into a command for operating the display device 200.

[0037] The method of analyzing the voice data and converting it into a command is not particularly limited since it is not a feature of this embodiment. A common method is to convert the voice data into text using a known voice recognition technology and obtain a command that matches the text data from a list of pre-registered commands.

[0038] The voice analysis device 5 operates the display device 200 by transmitting command information obtained by analyzing the voice data to the display device 200. The command information transmitted at this time includes associated data such as parameters required for command execution. Although the command transmission means is not particularly limited in this embodiment, there is a method in which a communication session is established in advance from the display device 200 to the information processing system 100. The display device 200 can communicate with the information processing system 100 from inside the firewall, and thereafter, by maintaining the communication session, information can be received from the information processing system 100. The voice analysis device 5 communicates with the display device 200 using this communication session, and uses a WebAPI published by the display device 200.

[0039] Fig. 2 is a diagram for explaining the operation of a user holding an IC card over display device 200. The example of Fig. 2 shows a case where a user holds an IC card 3 over display device 200. The user may also hold a terminal device 4 with an IC card function over display device 200.

[0040] The display device 200 has a short-distance communication unit 219. This short-distance communication unit 219 is an IC card reader. In this embodiment, first, the user moves to the location where the display device 200 is installed, and brings an IC card that the user possesses into close proximity to the short-distance communication unit 219 of the display device 200.

[0041] Then, the short-range communication unit 219 of the display device 200 acquires a card ID for identifying the IC card from the IC card 3, and transmits it to the information processing system 100. The card ID is, for example, a unique ID for identifying the IC card, or a number of 8 to 16 characters consisting of the numbers 0 to 9 and the letters A to F. Specifically, for example, if the IC card is a MIFERA (registered trademark) card, the card ID is a number called UiD, and if the IC card is a Felica (registered trademark) card, the card ID is a number called IDm.

[0042] Furthermore, the IC card 3 of the present embodiment may be capable of writing an email address or a user ID. In this case, the display device 200 transmits the email address (which has the function of a user ID due to its uniqueness) or the user ID written in the IC card 3 to the information processing system 100.

[0043] <Hardware configuration example> Hereinafter, the hardware configurations of the device, the information processing system 100, the terminal device 4, and the IC card 3 described in this embodiment will be described with reference to FIGS.

[0044] <<Equipment>> Fig. 3 is a diagram showing an example of a hardware configuration of the display device 200. In Fig. 3, the display device 200 will be described as an example of a device. The display device 200 of this embodiment includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an SSD (Solid State Drive) 204, a network controller 205, and an external device connection I / F 206 (Interface), and is a shared terminal for sharing information among a plurality of users.

[0045] Of these, the CPU 201 controls the overall operation of the display device 200. The ROM 202 stores programs used to drive the CPU 201, such as the CPU 201 and an IPL (Initial Program Loader). The RAM 203 is used as a work area for the CPU 201. The SSD 204 stores various data such as programs for the display device. The network controller 205 controls communication with the network. The external device connection I / F 206 controls communication with a USB (Universal Serial Bus) memory 2600 and external devices (a camera 2400, a speaker 2300, and a microphone 2200).

[0046] The display device 200 also includes a capture device 211, a GPU 212, a display controller 213, a contact sensor 214, a sensor controller 215, an electronic pen controller 216, a short-range communication unit 219, and an antenna 219a of the short-range communication unit 219.

[0047] Of these, the capture device 211 displays video information as still images or moving images on the display of the PC 10. The GPU (Graphics Processing Unit) 212 is a semiconductor chip that specializes in graphics. The display controller 213 controls and manages the screen display in order to output the output image from the GPU 212 to the display 220 etc.

[0048] The contact sensor 214 detects that the electronic pen 2500, the user's hand H, or the like has come into contact with the display 220. The sensor controller 215 controls the processing of the contact sensor 214. The contact sensor 214 inputs and detects coordinates using an infrared blocking method. The method of inputting and detecting coordinates may be a method in which two light receiving and emitting devices installed at both ends of the upper side of the display 220 emit a plurality of infrared rays in parallel to the display 220, and receive light that is reflected by a reflecting member installed around the display 220 and returns along the same optical path as the optical path of the light emitted by the light receiving element.

[0049] The contact sensor 214 outputs the ID of the infrared rays radiated by the two light receiving and emitting devices that are blocked by the object to the sensor controller 215, and the sensor controller 215 specifies the coordinate position that is the contact position of the object. The electronic pen controller 216 communicates with the electronic pen 2500 to determine whether the pen tip or the pen tail has touched the display 220. The near field communication unit 219 is a communication circuit such as NFC or Bluetooth (registered trademark).

[0050] Furthermore, the display device 200 includes a bus line 210. The bus line 210 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 201.

[0051] The contact sensor 214 is not limited to an infrared blocking type, and may use various detection means such as a capacitive touch panel that identifies the contact position by detecting a change in capacitance, a resistive film touch panel that identifies the contact position by a voltage change between two opposing resistive films, or an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by a contacting object touching the display unit. The electronic pen controller 216 may determine whether or not a part of the electronic pen 2500 that is held by the user or other parts of the electronic pen has been touched, in addition to the tip and bottom of the electronic pen 2500.

[0052] <<Information Processing System>> 4 is a diagram showing an example of a hardware configuration of the information processing system 100. The information processing system 100 includes a CPU 301, a ROM 302, a RAM 303, a hard disk (HD) 304, a hard disk drive (HDD) 305, a recording medium 306, a media I / F 307, a display 308, a network I / F 309, a keyboard 311, a mouse 312, a compact disc read only memory (CD-ROM) drive 314, and a bus line 310.

[0053] Of these, the CPU 301 controls the overall operation of the information processing system 100. The ROM 302 stores programs used to drive the CPU 301, such as the IPL. The RAM 303 is used as a work area for the CPU 301. The HD 304 stores various data such as programs. The HDD 305 controls the reading and writing of various data from and to the HD 304 under the control of the CPU 301.

[0054] The media I / F 307 controls reading and writing (storing) of data from and to a recording medium 306 such as a flash memory. The display 308 displays various information such as a cursor, a menu, a window, characters, or an image. The network I / F 309 is an interface for data communication using a network.

[0055] The keyboard 311 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. The mouse 312 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The CD-ROM drive 314 controls reading and writing of various data from a CD-ROM 313, which is an example of a removable recording medium. The bus line 310 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 301 shown in FIG. 4.

[0056] <<ICカード> > 5 is a diagram showing an example of the hardware configuration of an IC card. Here, a non-contact type configuration will be described, but an IC card 3 that may be a contact type has an IC chip 501 and an antenna coil 507. Furthermore, the IC chip 501 has a CPU 502, a ROM 503, a RAM 505, an EEPROM (Electrically Erasable and Programmable ROM) 504, and an antenna I / F 508.

[0057] Of these, the CPU 502 controls the overall operation of the IC card 3. The ROM 503 stores programs used to drive the CPU 502. The RAM 505 is used as a work area for the CPU 502. The EEPROM 504 stores programs for the IC card 3 and various data such as a terminal ID for identifying the IC card 3. The antenna I / F 508 controls transmission and reception of data to and from an external device via an antenna coil 507.

[0058] Furthermore, the IC card 3 includes a bus line 506. The bus line 506 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 502.

[0059] The antenna coil 507 is a coil for generating a current upon receiving the magnetic field generated by an external device when the IC card 3 passes through the magnetic field. By using this electricity, the IC card 3 can activate the IC chip 501, communicate with a reader / writer of the external device, and obtain and provide data.

[0060] <<Terminal Device>> Fig. 6 is a diagram showing an example of a hardware configuration of the terminal device 4. In Fig. 6, a smartphone is taken as an example of the terminal device 4. The terminal device 4 includes a CPU 401, a ROM 402, a RAM 403, an EEPROM 404, a CMOS (Complementary Metal Oxide Semiconductor) sensor 405, an acceleration / direction sensor 406, a media I / F 408, and a GPS receiving unit 409.

[0061] Of these, the CPU 401 controls the overall operation of the terminal device 4. The ROM 402 stores the CPU 401 and programs used to drive the CPU 401, such as the IPL. The RAM 403 is used as a work area for the CPU 401. The EEPROM 404 reads or writes various data, such as programs (applications) for the terminal device, under the control of the CPU 401.

[0062] The CMOS sensor 405 captures an image of a subject (mainly a self-portrait) and obtains image data under the control of the CPU 401. The acceleration / direction sensor 406 is an electronic magnetic compass that detects geomagnetism, a gyrocompass, an acceleration sensor, or other such sensor. The media I / F 408 controls the reading and writing (storage) of data from and to a recording medium 407 such as a flash memory. The GPS receiver 409 receives GPS signals from GPS satellites.

[0063] The terminal device 4 also has a long-distance communication unit 411, a camera 412, an image sensor I / F 413, a microphone 414, a speaker 415, an audio input / output I / F 416, a display 417, an external device connection I / F 418, a short-distance communication unit 419, an antenna 419a of the short-distance communication unit 419, and a touch panel 421.

[0064] Among these, long-distance communication unit 411 is a circuit that communicates with other devices via a network. Camera 412 is a type of built-in imaging means that captures an image of a subject and obtains image data under the control of CPU 401. Image sensor I / F 413 is a circuit that controls the driving of camera 412. Microphone 414 is a type of built-in sound collection means that inputs sound.

[0065] The audio input / output I / F 416 is a circuit that processes input and output of audio signals between the microphone 414 and the speaker 415 under the control of the CPU 401. The display 417 is a type of display means such as liquid crystal or organic EL that displays an image of a subject, various icons, etc. The external device connection I / F 418 is an interface for connecting various external devices. The short-range communication unit 419 is a communication circuit such as NFC or Bluetooth (registered trademark). The touch panel 421 is a type of input means that allows a user to operate the terminal device 4 by pressing the display 417.

[0066] The terminal device 4 also includes a bus line 410. The bus line 410 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 401 shown in FIG.

[0067] <About the function> Next, functions of the terminal device 4, the information processing system 100, and the display device 200 will be described with reference to Fig. 7. Fig. 7 is an example of a functional block diagram in which functions of the terminal device 4, the information processing system 100, and the display device 200 are divided into blocks and described.

[0068] <<Terminal Device>> The terminal device 4 has a first communication unit 11 and a voice data acquisition unit 12. Each function of the terminal device 4 is a function or means realized by any of the components shown in Fig. 6 operating in response to an instruction from the CPU 401 in accordance with a program loaded from the EEPROM 404 onto the RAM 403. This program is, for example, an application dedicated to the voice operation system 1.

[0069] The first communication unit 11 communicates various information with the information processing system 100. In this embodiment, the first communication unit 11 transmits to the information processing system 100 voice data obtained by converting a voice spoken by a user into data, a user ID, and a time of speech.

[0070] The audio data acquisition unit 12 converts the audio input from the microphone 414 into audio data by PCM (pulse code modulation) encoding. The audio data may be a WAVE file or an MP3 file.

[0071] <<Information Processing System>> The information processing system 100 has a second communication unit 13, a function possessed by the voice analysis device 5, and a function possessed by the schedule management device 6. Each function possessed by the information processing system 100 is a function or means realized by any of the components shown in Fig. 4 operating by an instruction from the CPU 301 in accordance with a program loaded from the HD 304 onto the RAM 303.

[0072] First, the second communication unit 13 transmits and receives various information to and from the terminal device 4 and the display device 200. In this embodiment, the second communication unit 13 receives voice data, a user ID, and a speech time from the terminal device 4. The second communication unit 13 receives entry information from the display device 200, and transmits a command to the display device 200.

[0073] The voice analysis device 5 has a voice recognition unit 14, a command conversion unit 15, and a judgment unit 16. The voice recognition unit 14 extracts voice features from the PCM-encoded voice data, identifies a phoneme model, and identifies words using a pronunciation dictionary, and outputs text data of the identified words. Note that any method may be used for voice recognition. For example, a voice recognition method using an RNN (Recurrent Neural Network) is known.

[0074] As an example, the command conversion unit 15 extracts words by performing morphological analysis on the voice-recognized text data, and obtains commands that match the words from a list of commands pre-registered in the command information storage unit 17.

[0075] 8 shows an example of command information stored in the command information storage unit 17. Each item of the command information will be described.

[0076] A word is a word that is converted into a command. The words in the voice data are compared with the words in the command information. Note that the words in {} must be found as a set in the voice data.

[0077] A command is a command that is invoked by voice data.

[0078] The pre-entry permission indicates whether the user can execute a command before entering the conference room 7. This allows the user to turn on the power of devices and lights before entering the conference room 7.

[0079] For example, the command conversion unit 15 detects the words "power" and "on" from the voice data "power on" and identifies the command "power on" associated with "power, on." The command conversion unit 15 generates command information in a JSON format or an XML format, for example, {"command":"power on"}.

[0080] Returning to FIG. 7, the explanation will be given. The judgment unit 16 acquires schedule information from the schedule management device 6 and judges whether the speaker is a participant of the conference and has actually entered the conference room 7. From the schedule information, the judgment unit 16 identifies a conference that is currently being held or a conference that will be held within about 10 minutes from now. Then, the judgment unit 16 judges whether the speaker is registered as a participant in the conference and has already entered the conference. The current time may be the time of the utterance transmitted by the terminal device 4, or the current time (time of receiving the voice data) held by the voice analysis device 5.

[0081] The schedule management device 6 has a schedule management unit 18. The schedule management unit 18 manages the schedule information stored in a schedule information storage unit 19.

[0082] Fig. 9 is an example of schedule information stored in the schedule information storage unit 19. In detail, the information in Fig. 9 is obtained by extracting the meeting information from the schedule information. Each item of the meeting information will be described below.

[0083] - User ID is the identification information of the user participating in the conference.

[0084] The start time of the meeting is the reservation start time of conference room 7, and the end time is the reservation end time of conference room 7.

[0085] The meeting name is the name of the meeting, for example indicating what type of meeting it is.

[0086] The organizer may be, for example, the identification information of the user who has reserved the conference room 7, or the identification information of the user designated as the organizer.

[0087] Participants are users who plan to participate in a conference. For example, a conference is notified by email, and users who accept the invitation are registered.

[0088] The location is identification information of the conference room, such as the name of conference room 7.

[0089] The entry status is information on whether or not a user identified by a user ID has entered the conference room 7. When the user brings an IC card close to the short-range communication unit 219 of the display device 200 and the display device 200 transmits the user ID to the information processing system 100, the user is considered to have entered the conference room.

[0090] Materials are materials used in a conference. Each participant can register materials to be used in the conference in advance in the schedule information. Any participant in the conference can display the materials. Alternatively, for example, the materials may be restricted so that they can be displayed only by the participants who have registered them.

[0091] <<Display device>> Returning to Fig. 7, the display device 200 has a third communication unit 21, a command execution unit 22, and a display control unit 23. Each function of the terminal device 4 is a function or means realized by any of the components shown in Fig. 3 operating in response to an instruction from the CPU 201 in accordance with a program loaded from the SSD 204 onto the RAM 203.

[0092] The third communication unit 21 transmits the user ID received from the IC card to the information processing system 100, and also receives commands from the information processing system 100. As described above, after the display device 200 is started up, the third communication unit 21 maintains a communication session with the information processing system 100. For example, the third communication unit 21 may inquire of the information processing system 100 about the presence or absence of commands by polling, or may maintain the communication session by WebSocket.

[0093] The command execution unit 22 executes commands transmitted from the voice analysis device 5. The contents of the commands vary, but may be, for example, turning the display device 200 on or off, displaying text data, displaying graphics, connecting to the Internet, displaying materials, etc. The command execution unit 22 can directly acquire the voices of the participants from the microphone 2200, transmit them to the voice analysis device 5, and execute the commands received from the voice analysis device 5.

[0094] The display control unit 23 displays handwritten data, text data, images, menu buttons, and the like on the display 220.

[0095] 7 does not include general functions of the display device 200. In addition to those shown in the figure, the display device 200 may also include a contact position detection unit that detects the contact position of an input means, a drawing data generation unit that connects the contact positions to generate stroke data, a character recognition unit that recognizes handwritten data and converts it into text data, and the like.

[0096] <Processing Procedure> Next, the operation of the voice operation system 1 will be described with reference to Fig. 10. Fig. 10 is a sequence diagram illustrating the operation of the voice operation system 1.

[0097] S1: First, when a user enters the conference room 7, he / she holds the IC card 3 over the short-range communication unit 219 of the display device 200. The short-range communication unit 219 of the display device 200 receives the user ID and the like from the IC card 3. As a result, the display device 200 detects that the user has entered the conference room 7.

[0098] S2: The third communication unit 21 of the display device 200 transmits entry information to the schedule management device 6 of the information processing system 100, indicating that the user with the user ID read from the IC card 3 has entered the conference room 7. The third communication unit 21 may transmit information capable of identifying the conference room 7 that the user has entered, such as identification information for the conference room 7 or identification information for the display device 200. The second communication unit 13 of the schedule management device 6 receives the entry information, and the schedule management unit 18 identifies the conference room 7 based on the information capable of identifying the conference room 7, and identifies conference information being held in this conference room 7 at the current time or conference information being held within about 10 minutes from the current time. The schedule management unit 18 registers that the user has entered the conference information in which the user ID is registered as a participant (registers that the user has entered the conference room).

[0099] S3: Next, the user speaks to the terminal device 4 that the user owns in order to operate the display device 200 by voice. The user can operate the display device 200 by voice even before entering the room. A conference application is running on the terminal device 4, and when the user logs into the information processing system 100, a list of conferences in which the user is registered as a participant is displayed. The user selects a conference to attend and speaks. The voice data acquisition unit 12 of the terminal device 4 acquires the voice data.

[0100] S4: The first communication unit 11 of the terminal device 4 transmits the voice data, the user ID, and the speech time to the information processing system 100. The speech time is used to determine the order of priority when voice data is transmitted in parallel from a plurality of participants. Note that the speech time is not necessarily required.

[0101] 11 is a diagram illustrating information transmitted from the terminal device 4 to the information processing system 100. Voice data, a user ID, and a speech time are transmitted. The voice data includes the contents of the operation.

[0102] S5: Returning to Fig. 10, the second communication unit 13 of the information processing system 100 receives the voice data, the user ID, and the speech time. First, the determination unit 16 queries the schedule management unit 18 and acquires schedule information (i.e., meeting information) of the user ID transmitted together with the voice data.

[0103] S6: The voice recognition unit 14 analyzes the voice data and converts it into text data. The command conversion unit 15 converts the text data into a command via the second communication unit 13.

[0104] S7: Next, the decision unit 16 decides whether or not to permit the speaker to perform voice operations based on the schedule information. The details of this decision will be described with reference to FIG.

[0105] If it is determined in step S7 that voice operation is not permitted, the voice recognition unit 14 discards the voice data and the command, and does not execute the processes in and after step S8.

[0106] S8: If it is determined in step S7 that the voice operation is permitted, the second communication unit 13 of the information processing system 100 transmits a command to the display device 200.

[0107] S9: The third communication unit 21 of the display device 200 receives the command, and the command execution unit 22 executes a process according to the command.

[0108] 12 is a diagram for explaining the details of a method for determining whether or not to permit voice operation by the determination unit 16. FIG 12 is a flowchart for explaining the details of the determination in step S7 in FIG 10.

[0109] The determination unit 16 determines whether the speaker is a participant of the conference based on the schedule information acquired in step S5 (S71). First, the determination unit 16 detects a conference in which the speech time is included between the start time and the end time of the conference, or a conference whose start time is within about 10 minutes from the speech time. The determination unit 16 determines whether the user ID transmitted together with the voice data is included in the participants of the detected conference.

[0110] It should be noted that step S71 may be omitted if the app functions to allow the user to transmit voice data only to conferences that the user can participate in.

[0111] If the user ID transmitted together with the voice data is included in the participants of the conference, the determination unit 16 determines whether the command converted from the voice data can be executed before entering the conference room based on the command information indicating whether it can be executed before entering the conference room (S73). If the command can be executed before entering the conference room, the command is transmitted to the display device 200 (step S8 in FIG. 10).

[0112] If the command is not executable before entering the room, the determination unit 16 determines whether or not the user with the user ID transmitted together with the voice data has already entered the room based on the entry status of the conference information (S74).

[0113] If the user has already entered the room, a command is transmitted to the display device 200 (step S8 in FIG. 10). This determination makes it possible to prevent the user from operating the display device 200 remotely when the user is not near the display device 200. For example, it is possible to restrict operations such as distributing materials to be used in a meeting by the information processing system 100 to participants and turning off the power of the display device 200.

[0114] If the user ID transmitted together with the voice data is not included in the participants of the conference, or if the user has not already entered the conference room, the determination unit 16 discards the voice data and the command (S72).

[0115] <Command execution example> 13 is a diagram showing an example of displaying a document by executing a command. A user who has entered the conference room 7 utters "display document." The display device 200 transmits voice data to the voice analysis device 5. The voice analysis device 5 converts the voice data into a command to display the document, and transmits the document data of the conference information registered as a participant by the user and the command to the display device 200. This enables the display device 200 to display the document on the display 220.

[0116] <Major Effects> As described above, in the voice operation system 1 of this embodiment, when the speaker is a participant of the conference, the voice analysis device 5 controls the devices in the conference room 7 based on the voice data of the speaker. Therefore, the people registered as participants of the conference can operate the devices in the conference room 7.

[0117] Furthermore, in the voice operation system 1, when the speaker is a participant of the conference room and is actually in the conference room 7, the voice analysis device 5 controls the devices in the conference room 7 based on the voice data of the speaker. Therefore, when the user is not near the display device 200, remote operation of the display device 200 can be suppressed. EXAMPLES

[0118] In this embodiment, a voice operation system 1 in which voice operation of a device is restricted by voice operation authority for each participant will be described.

[0119] In this embodiment, it is assumed that the hardware configuration diagrams of FIGS. 3 to 6 and the functional block diagram shown in FIG. 7 described in the above embodiment can be used.

[0120] Fig. 14 shows schedule information registered in the schedule management device 6 in this embodiment. In addition to the schedule information of the first embodiment shown in Fig. 9, the schedule information shown in Fig. 14 includes a setting as to whether or not voice operation authority is to be granted to each user of each conference. Regarding a method of setting voice operation authority, for example, the organizer of each conference may be able to set the presence or absence of voice operation authority for each participant. Alternatively, for example, voice operation authority may be automatically granted only to the organizer. The method of setting voice operation authority is not limited to the above.

[0121] Fig. 15 is a flowchart for explaining the details of the determination in step S7 in Fig. 10. Note that the explanation of Fig. 15 will mainly focus on the differences from Fig. 12. The flowchart of Fig. 15 newly includes step S71-2.

[0122] If the speaker is a conference participant (Yes in S71), the determination unit 16 determines whether the speaker has voice operation authority (S71-2). That is, the determination unit 16 determines whether the speaker's user ID in the conference information has voice operation authority.

[0123] If the speaker has voice operation authority, the process proceeds to step S73; if not, the process proceeds to step S72.

[0124] <Major Effects> According to the voice operation system 1 of this embodiment, in addition to the effect of the first embodiment, the organizer can grant the authority to operate the device to only some of the participants of the conference. In other words, the voice operation system 1 can restrict the participants from operating the device.

[0125] <Other application examples> The above describes the best mode for carrying out the present invention using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0126] For example, in the present embodiment, an example has been described in which a user moves to a conference room, but the user may also participate online. There may be one or more participants in a conference room. When participating in a conference online, the presence or absence of entry to the conference is determined by the user logging in to the information processing system 100. If a user logs in within about 10 minutes of the start time of the conference, the information processing system 100 determines that the user has entered the conference in which he or she is a participant.

[0127] In addition, in the present embodiment, a display device that can be used as an electronic whiteboard has been described, but the display device may be any device capable of displaying images, such as digital signage, etc. Also, a projector may be used instead of a display.

[0128] A device having the same function as an electronic whiteboard is also called an electronic whiteboard, an electronic information board, an interactive board, etc. Other information processing devices may be, for example, a PJ (Projector), an output device such as digital signage, a HUD (Head Up Display) device, an industrial machine, an imaging device, a sound collecting device, a medical device, a network home appliance, a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a game machine, a PDA, a digital camera, a wearable PC, a desktop PC, etc.

[0129] In addition, the configuration example in Fig. 7 and the like is divided according to main functions in order to facilitate understanding of the processing by the display device 200. The present invention is not limited by the manner in which the processing units are divided or the names of the processing units. The processing of the display device 200 can be divided into more processing units according to the processing contents. Also, it can be divided so that one processing unit includes more processes.

[0130] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above. [Explanation of symbols]

[0131] 1. Voice control system 4 Terminal Equipment 100 Information Processing Systems 200 Display device [Prior art documents] [Patent documents]

[0132] [Patent Document 1] JP 2019-145090 A

Claims

1. An information processing system that operates equipment used in a conference based on voice data transmitted from a terminal device, a communication unit that receives user identification information and voice data held by the terminal device from the terminal device; a command conversion unit that converts the voice data into a command related to an operation of the device; A determination unit that determines whether or not the information on the participants of the conference includes identification information of the user, When the determination unit determines that the information about the participants of the conference includes identification information of the user, the determination unit further determines whether or not the information about the participants of the conference includes information that the user has entered a conference room; When the user has entered the conference room, the communication unit transmits the command to the device.

2. When the determination unit determines that the information about the conference participants includes identification information of the user, the determination unit further determines whether or not the information about the conference participants indicates that the user has voice operation authority; The information processing system according to claim 1 , wherein the communication unit transmits the command to the device when the voice operation authority is set for the user and the user has already entered the conference room.

3. The command regarding the operation of the device is set to be executable before the user enters the conference room, If the command converted by the command conversion unit is executable before the user enters the conference room, 3. The information processing system according to claim 1, wherein the communication unit transmits the command to the device even if the user has not yet entered the conference room.

4. the communication unit receives entry information indicating that the user has entered the conference room from an input device provided at an entrance of the conference room or from the device used in the conference; 4. The information processing system according to claim 1, further comprising a schedule management unit that registers, in the information on the participants of the conference, that the user has entered the conference room.

5. 1. A voice operation system having an information processing system for operating a device used in a conference based on voice data transmitted from a terminal device, and the device, The information processing system includes: a communication unit that receives user identification information and voice data held by the terminal device from the terminal device; a command conversion unit that converts the voice data into a command related to an operation of the device; A determination unit that determines whether or not the information on the participants of the conference includes identification information of the user, When the determination unit determines that the information about the participants of the conference includes identification information of the user, the determination unit further determines whether or not the information about the participants of the conference includes information that the user has entered a conference room; If the user has entered the conference room, the communication unit transmits the command to the device; The device comprises: A voice operation system having a command execution unit that executes a command transmitted from the information processing system.

6. An information processing system for operating devices used in a conference based on voice data transmitted from a terminal device, a communication unit that receives user identification information and voice data held by the terminal device from the terminal device; a command conversion unit that converts the voice data into a command related to an operation of the device; a determining unit that determines whether or not the information about the participants of the conference includes identification information of the user; When the determination unit determines that the information about the participants of the conference includes identification information of the user, the determination unit further determines whether or not the information about the participants of the conference includes information that the user has entered a conference room; If the user has already entered the conference room, the communication unit transmits the command to the device.

7. 1. A voice operation method performed by an information processing system that operates equipment used in a conference based on voice data transmitted from a terminal device, comprising: receiving, from the terminal device, user identification information and voice data held by the terminal device; converting the voice data into commands related to the operation of the device; determining whether information about participants in a conference includes identification information of the user; If the information about the participants of the conference includes identification information of the user, the step of determining whether or not the information about the participants of the conference includes information that the user has entered the conference room; sending the command to the device if the user has entered the conference room; A voice operation method comprising:

Images