Information processing system, control method for information processing system, and program

The information processing system addresses the challenge of user familiarity with processing-utterance connections by displaying utterance examples linked to touch objects, improving user understanding through visual associations.

JP7581038B2Active Publication Date: 2024-11-12CANON KK
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
JP2020209345
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-12-17
Publication Date
2024-11-12
Estimated Expiration
2040-12-17

AI Technical Summary

Technical Problem

Users find it difficult to learn the connection between executed processing and utterance instructions in information processing systems due to the lack of semantic connection between numerical or alphabetic identifiers and the processing they represent.

Method used

An information processing system that includes a display device, microphone, and output means to display utterance examples associated with touch objects, managing word information in association with utterance examples, and executing processes linked to these objects, with filters for mismatched inputs.

Benefits of technology

Facilitates easy learning of the connection between executed processing and utterance instructions by visually associating words with touch objects, enhancing user familiarity and operational ease.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007581038000001
    Figure 0007581038000001
  • Figure 0007581038000002
    Figure 0007581038000002
  • Figure 0007581038000003
    Figure 0007581038000003
Patent Text Reader

Abstract

To provide an information processing system that allows a user to easily familiarize themselves with the connection between processing to be executed and an utterance instruction.SOLUTION: A server 102 transmits text data to an image forming apparatus 101 based on voice data acquired from a voice controller 100. The image forming apparatus 101 executes predetermined processing linked to UI components based on voice recognition information and the text data. The voice recognition information includes at least words related to the predetermined processing linked to the UI components, and is displayed in association with the UI components included in a screen being displayed on a touch panel 200 of the image forming apparatus 101.SELECTED DRAWING: Figure 11
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an information processing system. , love The present invention relates to a control method and a program for an information processing system. [Background technology]

[0002] Recently, information processing systems that analyze user utterances and execute commands corresponding to the utterances have become widespread. As a related technology, the technology of Patent Document 1 has been proposed. In Patent Document 1, identifiers such as "numbers" and "alphabet" are added to UI parts on the operation screen of an information processing device, and by accepting spoken instructions using these identifiers, processing corresponding to the UI parts is realized by speech. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-112933 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, such identifiers, such as numbers and letters, have no semantic association with the processes to be executed, and are difficult for users to become familiar with. For this reason, an information processing system is desired that allows users to easily become familiar with the association between the processes to be executed and spoken instructions.

[0005] The object of the present invention is to provide an information processing system that allows a user to easily learn the relationship between the processing to be executed and the spoken instructions. , love The present invention provides a control method and a program for an information processing system. [Means for solving the problem]

[0006] In order to achieve the above object, an information processing system of the present invention includes a display device capable of displaying information, a microphone capable of acquiring sound, an output means for outputting word information based on voice information in a natural language inputted through the microphone, and a display control means for additionally displaying example utterances in association with a touch object included in a screen being displayed on the display device; before An execution means for executing a predetermined process associated with the touch object; The utterance example includes at least a word constituting a process name of the predetermined process, and for each of a plurality of touch objects displayed on the display device, a command for executing the associated predetermined process, word information for a filter, and the utterance example are managed in association with each other, and when the outputted word information matches information of a combination of words included in the utterance example, the execution means executes the predetermined process associated with the touch object managed in association with the utterance example, and when the outputted word information does not match information of a combination of words included in the utterance example, the display control means displays on the display device a touch object managed in association with word information for a filter that matches the outputted word information, and the utterance example in association with the touch object. It is characterized by the above. Effect of the Invention

[0007] According to the present invention, a user can easily become familiar with the association between a process to be executed and a speech instruction. [Brief description of the drawings]

[0008] [Figure 1] 1 is a configuration diagram of an information processing system according to an embodiment of the present invention. [Diagram 2] 2 is a diagram showing an example of a home screen displayed on a touch panel of the operation panel in FIG. 1. [Diagram 3] 2 is a block diagram illustrating a schematic hardware configuration of the voice control device of FIG. 1. [Figure 4] 2 is a block diagram illustrating a schematic hardware configuration of a controller unit of the server in FIG. 1. [Diagram 5] FIG. 2 is a block diagram illustrating a schematic hardware configuration of the image forming apparatus in FIG. [Figure 6] 4 is a block diagram showing a functional configuration of a voice control program of the voice control device executed by a CPU of FIG. 3. [Figure 7] 5 is a block diagram showing a functional configuration of a voice recognition program of a server executed by a CPU of FIG. 4. [Figure 8A] 5 is a block diagram showing a functional configuration of a server remote control program executed by a CPU in FIG. 4. [Figure 8B] 8B is a diagram for explaining information managed by the data management unit in FIG. 8A. [Figure 9] 6 is a block diagram showing a functional configuration of a device control program of the image forming apparatus executed by a CPU of FIG. 5. [Figure 10] 10 is a sequence diagram showing the procedure of a voice operation control process executed by the information processing system according to the embodiment. FIG. [Figure 11] 3 is a diagram showing an example of a screen displayed on the touch panel of FIG. 2. FIG. [Figure 12] 10 is a flowchart showing the procedure of a screen update control process executed by the device control program of FIG. [Figure 13] 7 is a flowchart showing a procedure of a voice control process executed by the voice control program of FIG. 6. [Figure 14] 8 is a flowchart showing the procedure of a voice recognition control process executed by the voice recognition program of FIG. 7. [Figure 15] 8B is a flowchart showing a procedure of a remote control process executed by the remote control program of FIG. 8A. [Figure 16] 3 is a diagram showing an example of a home screen displayed on the touch panel of FIG. 2. [Figure 17] 11 is a diagram showing an example of a setting screen for setting voice recognition information and filter words in the present embodiment. FIG. [Figure 18] 8B is a flowchart showing a procedure of a setting control process executed by the remote control program of FIG. 8A. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the present invention will be described in detail with reference to the accompanying drawings, showing examples of the present invention. However, the components shown in the examples are merely illustrative and are not intended to limit the scope of the present invention.

[0010] Fig. 1 is a configuration diagram of an information processing system according to the present embodiment. As shown in Fig. 1, the information processing system is composed of a voice control device 100, an image forming device 101 (image processing device), a server 102 (information processing device), a client terminal 103, and a gateway 105. Note that the information processing system may be configured to include a plurality of voice control devices 100, image forming devices 101, and client terminals 103.

[0011] The voice control device 100, the image forming device 101, and the client terminal 103 can communicate with each other via a gateway 105 and a network 104. In addition, the voice control device 100, the image forming device 101, and the client terminal 103 can communicate with the server 102 via the gateway 105 and the Internet 107.

[0012] The voice control device 100 acquires the voice of the user 106 in accordance with an instruction to start a voice operation from the user 106, and transmits voice data obtained by encoding the acquired voice (voice information) to the server 102. The voice control device 100 also outputs the voice data received from the server 102 as voice. The voice control device 100 is a voice input / output device capable of communicating with the user by voice, such as a smart speaker or a smartphone. In this embodiment, the voice control device 100 and the image forming device 101 are configured independently. However, the hardware (hardware blocks described later using FIG. 3) and software functions (software blocks described later using FIG. 6) constituting the voice control device 100 may be included in the image forming device 101, and the configuration is not limited to this.

[0013] The image forming apparatus 101 is a multifunction machine having multiple functions such as a copy function, a scan function, a print function, and a FAX function, but may be a printer or a scanner having a single function. The following description is based on the assumption that the image forming apparatus 101 is a color laser beam multifunction machine. The image forming apparatus 101 also has an operation panel 108. The operation panel 108 is a display unit that displays a screen for accepting operation instructions from a user 106 and the status of the image forming apparatus 101. The operation panel 108 also has a touch panel 200 (display device) shown in FIG. 2, which will be described later, that is integrated with an LCD display, and also functions as an input unit that accepts operations from the user 106.

[0014] The server 102 performs voice recognition on the voice data of the user 106 acquired from the voice control device 100, and determines words related to the setting operation of the image forming device 101 and the execution of a job from the voice recognition result. Note that a job refers to a unit of a series of image forming processes (e.g., copy, scan, print, etc.) that the image forming device 101 realizes using a print engine 513 and a scanner 515 in FIG. 5 described later. The server 102 also generates text according to the voice recognition result or the word determination result, and synthesizes voice data for playing the content of the text aloud on the voice control device 100. Note that the server 102 can provide highly accurate voice recognition results by machine learning using a neural network such as deep learning. For example, the server 102 has been trained to accurately recognize voices from users far away. The server 102 also supports natural language processing, and can acquire appropriate information (words, kana-kanji conversion results) from input natural language through morphological analysis, syntactic analysis, semantic analysis, context analysis, etc.

[0015] The client terminal 103 is, for example, a personal computer (PC) used by a user 106. The client terminal 103 generates a print job for causing the image forming apparatus 101 to print an electronic file stored in an external storage device 505 of the image forming apparatus 101 (described later) in FIG. 5, the client terminal 103, or the like. The client terminal 103 also receives image data from the image forming apparatus 101 that the image forming apparatus 101 has generated by scanning.

[0016] The network 104 mutually connects the voice control device 100, the image forming device 101, the client terminal 103, and the gateway 105. The network 104 transmits and receives various data such as print jobs and scan jobs, and voice data.

[0017] The gateway 105 is, for example, a wireless LAN router conforming to the IEEE802.11 standard series. The IEEE802.11 standard series includes a series of standards belonging to IEEE802.11, such as IEEE802.11a and IEEE802.11b. The gateway 105 may have the capability of operating according to a wireless communication method other than the IEEE802.11 standard series. The gateway 105 may also be a wired LAN router conforming to Ethernet standards such as 10BASE-T, 100BASE-T, and 1000BASE-T, instead of a wireless LAN router, and may have the capability of operating according to another wired communication method.

[0018] 2 is a diagram showing an example of a home screen 201 displayed on touch panel 200 of operation panel 108 in FIG. 1. Home screen 201 is displayed when image forming apparatus 101 is started up. UI widgets (touch objects) for each function executed by image forming apparatus 101 are displayed on home screen 201. Here, a UI widget refers to a certain partitioned area (button, icon, mark, arrow, tab, rectangle) on touch panel 200 that is distinguishable by user 106. When user 106 touches these UI widgets, a function associated with the touched UI widget is executed.

[0019] The copy 202 is a UI component for transitioning to a copy screen 1112 in FIG. 11(b) described later, where settings necessary for executing a copy function are performed. The scan 203 is a UI component for transitioning to a scan screen (not shown) where settings necessary for executing a scan function are performed. The menu 204 is a UI component for transitioning to a menu screen (not shown) where settings such as the display language of the operation panel 108 are performed. The address book 205 is a UI component for transitioning to an address book screen where settings are made for the destination of image data generated by the image forming apparatus 101 through scanning. The secure print 206 is a UI component for transitioning to a print screen (not shown) for printing image data with a password received by the image forming apparatus 101. The voice recognition 207 is a UI component for enabling voice operation by the voice control apparatus 100. When the voice recognition 207 is pressed and voice operation by the voice control apparatus 100 is enabled, "voice recognition in progress" is displayed in the status display 210 area.

[0020] The status check 208 is a UI part for displaying information on jobs executed by the image forming apparatus 101 and jobs currently being executed. The tab area 209 displays tab numbers "1" to "7". The home screen 201 is composed of a plurality of page screens, and when the user 106 presses any of the tab numbers "2" to "7", the page screen is switched to the page screen corresponding to the pressed tab number. In this page screen, a UI part different from the UI part shown in FIG. 2 is displayed. With this configuration, the user can easily display another UI part that could not be displayed on the home screen 201 by a simple operation of pressing a tab number. As described above, the status display 210 displays "voice recognition in progress", which indicates that voice operation is possible. In addition, the status display 210 displays the job status such as "printing", "receiving", "sending", and "reading", and errors such as "no paper", "jam", and "no toner".

[0021] Fig. 3 is a block diagram showing a schematic hardware configuration of the voice control device 100 of Fig. 1. In Fig. 3, the voice control device 100 includes a controller unit 300, a microphone 308, a speaker 310 (voice output device), and an LED 312. The controller unit 300 is connected to the microphone 308, the speaker 310, and the LED 312. The controller unit 300 also includes a CPU 302, a RAM 303, a ROM 304, an external storage device 305, a network I / F 306, a microphone I / F 307, an audio controller 309, and a display controller 311. The CPU 302, the RAM 303, the ROM 304, the external storage device 305, the network I / F 306, the microphone I / F 307, the audio controller 309, and the display controller 311 are connected to each other via a system bus 301.

[0022] The CPU 302 is a central processing unit that controls the operation of the entire controller unit 300. The RAM 303 is a volatile memory. The ROM 304 is a non-volatile memory, and stores a startup program for the CPU 302. The external storage device 305 is a storage device (e.g., an SD card) with a larger capacity than the RAM 303. The external storage device 305 may be a storage device other than an SD card, such as a flash ROM, as long as it has the same functions as an SD card. The external storage device 305 stores a control program for the voice control device 100 executed by the controller unit 300, such as a voice control program 601 in FIG. 6, which will be described later.

[0023] When the CPU 302 is started by turning on the power, the CPU 302 executes a startup program stored in the ROM 304. The startup program is a program for reading out a control program stored in the external storage device 305 and expanding the control program on the RAM 303. After executing the startup program, the CPU 302 subsequently executes the control program expanded on the RAM 303, and performs audio input / output control, display control, and data communication control with the network 104. The CPU 302 also stores data used when executing the control program on the RAM 303 and reads and writes the data. Various settings and the like required when executing the control program can be stored on the external storage device 305. The various settings are, for example, the URL of the server 102 that enables access to the image forming apparatus 101, and are read and written by the CPU 302. The CPU 302 communicates with other devices on the network 104 via the network I / F 306.

[0024] The network I / F 306 includes a circuit and an antenna for performing communication according to a wireless communication method compliant with the IEEE802.11 standard series. The communication method of the network I / F 306 is not limited to a wireless communication method, and may be, for example, a wired communication method compliant with the Ethernet standard. A microphone 308 is connected to the microphone I / F 307. The microphone I / F 307 acquires a voice uttered by the user 106 from the microphone 308, encodes the acquired voice, converts it into voice data, and stores the voice data in the RAM 303 according to an instruction from the CPU 302.

[0025] The microphone 308 is a device for voice operation capable of acquiring the voice of the user 106, and is, for example, a small MEMS microphone mounted on a smartphone or the like. It is preferable to use three or more microphones 308 arranged at predetermined positions so that the direction of arrival of the voice emitted by the user 106 can be calculated. Note that the present embodiment can be realized even with one microphone 308, and three or more microphones are not essential. A speaker 310 is connected to the audio controller 309. The audio controller 309 converts audio data into an analog audio signal according to an instruction from the CPU 302, and outputs the audio through the speaker 310.

[0026] The speaker 310 is a general-purpose device for playing back audio. The speaker 310 plays back the response sound of the device that indicates that the audio control device 100 is responding, and the audio synthesized by the server 102.

[0027] The display controller 311 is connected to an LED 312. The display controller 311 controls the display of the LED 312 according to an instruction from the CPU 302. For example, the display controller 311 controls the lighting of an LED to indicate that the voice control device 100 is correctly acquiring the voice of the user 106. The LED 312 is, for example, a blue LED that is visible to the user 106. The LED 312 is a general-purpose device. The LED 312 may be replaced with a display device capable of displaying characters and pictures.

[0028] Fig. 4 is a block diagram showing a schematic hardware configuration of the controller unit of the server 102 of Fig. 1. As shown in Fig. 4, the server 102 includes a controller unit 400a and a controller unit 400b. In the present embodiment, the controller units 400a and 400b have the same configuration, and the configuration will be described below using the controller unit 400a as an example. The controller unit 400a executes a voice recognition program 701 of Fig. 7, which will be described later. The controller unit 400a includes a CPU 402a, a RAM 403a, a ROM 404a, an external storage device 405a, and a network I / F 406a, all connected to a system bus 401a.

[0029] The CPU 402a is a central processing unit that controls the operation of the entire controller unit 400a. The RAM 403a is a volatile memory. The ROM 404a is a non-volatile memory, and stores a startup program for the CPU 402a. The external storage device 405a is a storage device (for example, a hard disk drive: HDD) having a larger capacity than the RAM 403a. The external storage device 405a stores a control program for the server 102 executed by the controller unit 400a, for example, a voice recognition program 701 in FIG. 7, which will be described later. The external storage device 405a may be a storage device other than a hard disk drive, for example, a solid state drive (SSD), as long as the storage device has a function equivalent to that of a hard disk drive. The external storage device 405a may also be an external storage accessible as the server 102.

[0030] At the time of startup, such as when the power is turned on, the CPU 402a executes a startup program stored in the ROM 404a. This startup program is for reading out a control program stored in the external storage device 405a and expanding the control program on the RAM 403a. After executing the startup program, the CPU 402a subsequently executes the control program expanded on the RAM 403a to perform control. The CPU 402a also stores data used when executing the control program on the RAM 403a and reads and writes it. Various settings required when executing the control program can also be stored on the external storage device 405a. The various settings are read and written by the CPU 402a. The CPU 402a communicates with other devices on the network 104 via the network I / F 406a.

[0031] The controller unit 400b executes a remote control program 801 in FIG. 8, which will be described later. The controller unit 400b includes a CPU 402b, a RAM 403b, a ROM 404b, an external storage device 405b, and a network I / F 406b, which are connected to a system bus 401b. The CPU 402b, the RAM 403b, the ROM 404b, the external storage device 405b, and the network I / F 406b have the same functions and configurations as the CPU 402a, the RAM 403a, the ROM 404a, the external storage device 405a, and the network I / F 406a, respectively. The external storage device 405b stores a control program for the server 102 executed by the controller unit 400b, for example, a remote control program 801 in FIG. 8, which will be described later. In this embodiment, a configuration in which a voice recognition program 701 and a remote control program 801, which will be described later, are executed by different controller units will be described, but the present invention is not limited to this configuration. For example, either the controller unit 400a or the controller unit 400b may execute both the voice recognition program 701 and the remote control program 801, which will be described later.

[0032] 5 is a block diagram showing a schematic configuration of hardware of the image forming apparatus 101 shown in FIG. 1. As shown in FIG. 5, the image forming apparatus 101 includes a controller unit 500, an operation panel 108, a print engine 513 (printing device), and a scanner 515 (reading device). The controller unit 500 is connected to the operation panel 108, the print engine 513, and the scanner 515. The controller unit 500 also includes a CPU 502, a RAM 503, a ROM 504, an external storage device 505, a network I / F 506, a display controller 507, an operation I / F 508, a print controller 512, and a scan controller 514. The CPU 502, the RAM 503, the ROM 504, the external storage device 505, the network I / F 506, the display controller 507, the operation I / F 508, the print controller 512, and the scan controller 514 are connected to each other via a system bus 501.

[0033] The CPU 502 is a central processing unit that controls the operation of the entire controller unit 500. The RAM 503 is a volatile memory. The ROM 504 is a non-volatile memory that stores a startup program for the CPU 502. The external storage device 505 is a storage device (e.g., a hard disk drive: HDD) having a larger capacity than the RAM 503. The external storage device 505 stores a control program for the image forming apparatus 101 executed by the CPU 502, for example, a device control program 901 shown in FIG. 9, which will be described later. The external storage device 505 may be a storage device other than a hard disk drive, for example, a solid state drive (SSD), as long as it has the same functions as a hard disk drive.

[0034] At the time of start-up, such as when the power is turned on, the CPU 502 executes a start-up program stored in the ROM 504. This start-up program is for reading out a control program stored in the external storage device 505 and expanding the control program on the RAM 503. After executing the start-up program, the CPU 502 subsequently executes the control program expanded on the RAM 503 and performs control. The CPU 502 also stores data used when executing the control program on the RAM 503 and reads and writes the data. The external storage device 505 can further store various settings required when executing the control program and image data read by the scanner 515, which are read and written by the CPU 502. The CPU 502 communicates with other devices on the network 104 via the network I / F 506 and with the server 102 on the Internet via a gateway.

[0035] The operation panel 108 is connected to the display controller 507 and the operation I / F 508. The display controller 507 controls the screen display of the touch panel 200 of the operation panel 108 in accordance with instructions from the CPU 502. The operation I / F 508 inputs and outputs operation signals. When the touch panel 200 is pressed, the CPU 502 obtains, via the operation I / F 508, coordinates indicating the position where the touch panel 200 is pressed.

[0036] A print engine 513 is connected to the print controller 512. The print controller 512 transmits control commands and image data to the print engine 513 in accordance with instructions from the CPU 502. The print engine 513 performs a print process of printing the received image data on a sheet in accordance with the control command received from the print controller 512. A scanner 515 is connected to the scan controller 514. The scan controller 514 transmits control commands to the scanner 515 in accordance with instructions from the CPU 502, and writes image data received from the scanner 515 into the RAM 503. The scanner 515 performs a read process of reading an original on a platen glass (not shown) provided in the image forming apparatus 101 using an optical unit in accordance with the control command received from the scan controller 514.

[0037] Fig. 6 is a block diagram showing a functional configuration of a voice control program 601 of the voice control device 100 executed by the CPU 302 of Fig. 3. The voice control program 601 is stored in the external storage device 305 as described above, and the CPU 302 loads and executes the voice control program 601 on the RAM 303 at startup. The voice control program 601 is composed of a data transmission / reception unit 602, a data management unit 603, a voice acquisition unit 604, a voice playback unit 605, a display unit 606, a voice operation start detection unit 607, an utterance end determination unit 608, and a voice control unit 609.

[0038] The data transmission / reception unit 602 transmits and receives data to and from other devices on the network 104 using TCP / IP via the network I / F 306. The data transmission / reception unit 602 transmits voice data acquired from the voice acquisition unit 604 to the server 102. The data transmission / reception unit 602 also receives voice synthesis data (response to the user 106) generated on the server 102.

[0039] The data management unit 603 stores various data, such as operation data generated during execution of the voice control program 601, in a predetermined area on the external storage device 305 and manages them. For example, volume setting data for the voice played by the voice playback unit 605, authentication information required for communication with the gateway 105, device information required for communication with the image forming apparatus 101 and the server 102, program URLs, etc. are stored and managed.

[0040] The voice acquisition unit 604 converts the analog voice of the user 106 in the vicinity of the voice control device 100, acquired by the microphone 308, into voice data, and temporarily stores the voice data in the RAM 303. The voice of the user 106 is converted into a predetermined format such as MP3, and further encoded for transmission to the server 102, and temporarily stored as voice data in the RAM 303. The timing of the start and end of processing by the voice acquisition unit 604 is managed by the voice control unit 609. Note that the voice data may be encoded in a general-purpose streaming format, and the encoded voice data may be sequentially transmitted by the data transmission / reception unit 602.

[0041] The audio playback unit 605 plays the synthesized voice data received by the data transmission / reception unit 602 on the speaker 310 via the audio controller 309. The timing of audio playback by the audio playback unit 605 is managed by the audio control unit 609. The display unit 606 controls the lighting of the LED 312 via the display controller 311. For example, when the voice operation start detection unit 607 detects that a voice operation has been performed, the display unit 606 lights up the LED 312. The timing of lighting of the display unit 606 is managed by the audio control unit 609.

[0042] When the voice operation start detection unit 607 detects that the user 106 has spoken a wake word, that an operation start key (not shown) of the voice control device 100 has been pressed, or that the data transmission / reception unit 602 has received a voice control start command, it transmits an operation start notification to the voice control unit 609. Here, the wake word is a voice word that is predetermined for starting voice operation of the voice control device 100. The voice operation start detection unit 607 constantly detects the wake word from the analog voice of the user 106 in the vicinity of the voice control device 100 acquired by the microphone 308. The user 106 can operate the image forming device 101 by speaking the wake word and then issuing an operation instruction.

[0043] The speech end determination unit 608 determines the timing of the end of the process by the voice acquisition unit 604. For example, when the voice of the user 106 is interrupted for a predetermined time (for example, 3 seconds), the speech end determination unit 608 determines that the speech of the user 106 has ended, and transmits a speech end notification to the voice control unit 609. The speech end may be determined from a predetermined word uttered by the user 106, rather than from a time when there is no speech (hereinafter, referred to as blank time). For example, when a predetermined word such as "yes," "no," "OK," "cancel," "end," "start," or "start" is detected, the speech end may be determined without waiting for a predetermined time. The speech end may be determined by the server 102, not by the voice control device 100, and the speech end may be determined from the meaning or context of the content of the user 106's speech.

[0044] The voice control unit 609 is the center of control, and controls the other modules in the voice control program 601 to operate in cooperation with each other. Specifically, the voice control unit 609 controls the start and end of processing of the voice acquisition unit 604, the voice reproduction unit 605, and the display unit 606. The voice control unit 609 also controls the data transmission / reception unit 602 to transmit the voice data to the server 102 after the voice acquisition unit 604 converts the voice acquired by the microphone 308 into voice data. The voice control unit 609 also controls the voice reproduction unit 605 to reproduce the voice synthesis data after the data transmission / reception unit 602 receives voice synthesis data from the server 102.

[0045] Here, the timing of start and end of processing by the voice acquisition unit 604, the voice reproduction unit 605, and the display unit 606 will be described.

[0046] When the voice control unit 609 receives an operation start notification from the voice operation start detection unit 607, it starts the processing of the voice acquisition unit 604. When the voice control unit 609 receives an utterance end notification from the speech end determination unit 608, it ends the processing of the voice acquisition unit 604. For example, it is assumed that the user 106 utters a wake word and then utters "copy it". At this time, the voice operation start detection unit 607 detects the voice of the wake word and transmits an operation start notification to the voice control unit 609. When the voice control unit 609 receives the operation start notification, it controls the voice acquisition unit 604 to start the processing. The voice acquisition unit 604 converts the analog voice of "copy it" uttered following the wake word into voice data and temporarily stores the voice data. When the speech end determination unit 608 determines that there was a predetermined blank time after the utterance of "copy it", it transmits an utterance end notification to the voice control unit 609. When the voice control unit 609 receives the utterance end notification, it ends the processing of the voice acquisition unit 604. The state from when the speech acquisition unit 604 starts processing to when it ends is referred to as an utterance processing state. The display unit 606 turns on the LED 312 during the utterance processing state.

[0047] After determining that the user 106 has finished speaking, the voice control unit 609 controls the data transmission / reception unit 602 to transmit voice data to the server 102, and waits for a response from the server 102. The response from the server 102 includes, for example, a header indicating that it is a response, and voice synthesis data for reproducing the response message. When the voice control unit 609 receives the response from the server 102 via the data transmission / reception unit 602, it causes the voice reproduction unit 605 to reproduce the voice synthesis data included in the response from the server 102. The voice synthesis data is, for example, "Copy screen has been displayed." Note that the state from after the determination of the end of speaking to the end of reproduction of the voice synthesis data is referred to as a response processing state. The display unit 606 blinks the LED 312 during the response processing state.

[0048] After the response process, while the dialogue session with the server 102 continues, the user 106 can continue to speak what he or she wants to do without uttering the wake word. The server 102 determines whether to end the dialogue session by sending an end-of-dialogue session notification to the voice control device 100. The state from the end of a dialogue session to the start of the next dialogue session is called a standby state. The voice control device 100 remains in a standby state until it receives an operation start notification from the voice operation start detection unit 607. The display unit 606 turns off the LED 312 during the standby state.

[0049] Fig. 7 is a block diagram showing the functional configuration of a voice recognition program 701 of the server 102 executed by the CPU 402a of Fig. 4. The voice recognition program 701 is stored in the external storage device 405a as described above, and the CPU 402a loads the voice recognition program 701 on the RAM 403a at startup and executes it. The voice recognition program 701 is made up of a data transmission / reception unit 702, a data management unit 703, and a voice data conversion unit 704.

[0050] The data transmission / reception unit 702 transmits and receives data to and from other devices on the network 104 by TCP / IP via the network I / F 406a. The data transmission / reception unit 702 receives voice data of the user 106 from the voice control device 100.

[0051] The data management unit 703 stores various data, such as working data generated in the execution of the voice recognition program 701 and parameters required for the voice data conversion unit 704 to perform voice recognition processing, in a predetermined area on the external storage device 405a and manages them. For example, the data management unit 703 stores an acoustic model and a language model for converting voice data received by the data transmission / reception unit 702 into text (text information) in a predetermined area on the external storage device 405a and manages them. In addition, the data management unit 703 stores a dictionary for a morphological analysis unit 706 (described later) to perform morphological analysis of text in a predetermined area on the external storage device 405a and manages them. In addition, the data management unit 703 stores a voice database for a voice synthesis unit 707 (described later) to perform voice synthesis in a predetermined area on the external storage device 405a and manages them. Furthermore, the data management unit 703 stores and manages each device information required for communication with the voice control device 100, the image forming device 101, etc.

[0052] The voice data conversion unit 704 is composed of a voice recognition unit 705, a morphological analysis unit 706, and a voice synthesis unit 707. The voice recognition unit 705 performs a voice recognition process for converting the voice data of the user 106 received by the data transmission / reception unit 702 into text. The voice recognition process converts the voice data of the user 106 into phonemes using an acoustic model, and further converts the phonemes into actual text format data by pattern matching using a language model. The acoustic model may be a model using a machine learning method using a neural network such as DNN-HMM, or a model using a different method such as GMM-HMM. In machine learning using a neural network, for example, learning of a learning model is performed based on teacher data that pairs voice and text. The language model may be a model using a machine learning method using a neural network such as RNN, or a model using a different method such as the N-gram method.

[0053] In this embodiment, the text format data is composed of text consisting of one or more kana characters and text obtained by "kana-kanji conversion" (including conversion to numbers, alphabets, symbols, etc.). However, other methods may be used as a voice recognition process for converting voice data into text format data, and the method is not limited to the above-mentioned method.

[0054] The morpheme analysis unit 706 performs morpheme analysis on the text format data converted by the speech recognition unit 705. The morpheme analysis derives a morpheme string from a dictionary having information such as the grammar and parts of speech of the language, and further determines the part of speech of each morpheme. The morpheme analysis unit 706 can be realized by using known morpheme analysis software such as JUMAN, Chado, MeCab, etc. For example, the morpheme analysis unit 706 analyzes the text format data "I want to make a copy" converted by the speech recognition unit 705 as a morpheme string of "copy," "o," "shi," and "want to." Also, the morpheme analysis unit 706 analyzes the text format data "From A3 to A4" as a morpheme string of "A3," "From," "A4," and "To."

[0055] The voice synthesis unit 707 performs voice synthesis processing based on a notification received from the image forming apparatus 101. In the voice synthesis processing, in response to a predetermined notification, a previously prepared text paired with the predetermined notification is converted into voice synthesis data in a predetermined format such as MP3. In the voice synthesis processing, for example, voice synthesis data is generated based on a voice database stored in the data management unit 703. The voice database is a database that collects voices that utter fixed content such as words. Note that in this embodiment, the voice synthesis processing is performed using the voice database, but other methods may be used as a voice synthesis method and is not limited to the method using the voice database.

[0056] Fig. 8A is a block diagram showing the functional configuration of a remote control program 801 of the server 102 executed by the CPU 402b in Fig. 4. The remote control program 801 is stored in the external storage device 405b as described above, and the CPU 402b loads the remote control program 801 on the RAM 403b and executes it when started up. The remote control program 801 is made up of a data transmission / reception unit 802, a data management unit 803, and a remote control data conversion unit 804.

[0057] The data transmission / reception unit 802 transmits and receives data to and from other devices on the network 104 by TCP / IP via the network I / F 406b. The data transmission / reception unit 802 also receives text data (word information) obtained by performing morphological analysis from the voice recognition program 701. This text data is data in a text format consisting of one or more morphemes obtained by the voice recognition program 701 performing morphological analysis on the voice data of the user 106.

[0058] The data management unit 803 stores various data, such as parameters necessary for remotely controlling the image forming apparatus 101 based on the voice of the user 106 acquired by the voice control device 100, in a predetermined area on the external storage device 405b and manages them. Specifically, the data management unit 803 stores and manages the screen information, UI widget information, voice recognition information, operation instruction commands, and filter words shown in Fig. 8B.

[0059] The screen information is information indicating the type of screen, such as a home screen or a copy screen, displayed on touch panel 200 of image forming apparatus 101. In the present embodiment, with regard to a screen that has a plurality of page screens, such as the home screen, and that can be switched to each page by using tabs, the screen information of all page screens is managed by data management unit 803.

[0060] The UI widget information is information indicating UI widgets to be displayed on a screen indicated by the screen information. For example, names of UI widgets such as copy 202, scan 203, and menu 204 shown in Fig. 2, and tab numbers, etc. are managed as UI widget information. Note that in this embodiment, a case will be described in which names of UI widgets are managed as UI widget information, but this is not limited as long as the information indicates a UI widget. For example, the information may be managed by coordinate information indicating the position on the screen where the UI widget is displayed, or the display format of the UI widget (button, icon, input box, etc.).

[0061] The voice recognition information is a phrase for voice recognition of the UI widget indicated by the UI widget information. The voice recognition information includes at least words related to the process executed by the function corresponding to the UI widget, and has a semantic connection with the executed process. Different voice recognition information is set for each UI widget. Note that voice recognition information for the secure print 206, for which voice operation is not permitted from the viewpoint of security, is not set, and in FIG. 8B, this is indicated by "-" to indicate this. The remote control program 801 identifies the UI widget to be the target of voice operation based on the UI widget information and voice recognition information managed by the data management unit 803.

[0062] The operation instruction command is a command transmitted from the server 102 to the image forming apparatus 101 when the text data received from the voice recognition program 701 matches the voice recognition information, and a different command is associated with each piece of voice recognition information. The filter word is a word for filtering the display of UI components displayed on the touch panel 200, and a different filter word is set for each group to which the UI components belong. For example, the filter word "copy" is set for a UI component related to copying, and the filter word "scan" is set for a UI component related to scanning. If the filter word is the same as the voice recognition information, the filtered display cannot be performed. For this reason, the filter word is set to a word different from the voice recognition information, for example, a part of the words constituting the voice recognition information or another word having a similar meaning to the voice recognition information.

[0063] The remote control data conversion unit 804 is composed of a control command analysis unit 805, a voice character string data conversion unit 806, and a screen configuration acquisition unit 807. The control command analysis unit 805 compares the text data received from the voice recognition program 701 with a plurality of pieces of voice recognition information managed by the data management unit 803. If the plurality of pieces of voice recognition information includes voice recognition information that matches the text data, the remote control data conversion unit 804 transmits an operation instruction command corresponding to this voice recognition information to the image forming apparatus 101. On the other hand, if the plurality of pieces of voice recognition information does not include voice recognition information that matches the text data, the remote control data conversion unit 804 compares the text data with a plurality of filter words managed by the data management unit 803. If the plurality of filter words includes a filter word that matches the text data, the remote control data conversion unit 804 transmits the voice recognition information corresponding to this filter word and UI widget information corresponding to the voice recognition information to the image forming apparatus 101.

[0064] Upon receiving a screen update notification from the image forming apparatus 101 (described later), the voice character string data conversion unit 806 transmits text information of the voice data to be output from the speaker 310 of the voice control apparatus 100 to the voice recognition program 701 .

[0065] After transmitting the operation instruction command, the screen configuration acquisition unit 807 requests the data management unit 803 to store the screen information and UI widget information received from the image forming apparatus 101. In addition, the screen configuration acquisition unit 807 requests the data transmission / reception unit 802 to transmit to the image forming apparatus 101 voice recognition information to be displayed in association with the UI widgets included in the screen displayed on the touch panel 200 of the image forming apparatus 101.

[0066] Fig. 9 is a block diagram showing the functional configuration of a device control program 901 of the image forming apparatus 101 executed by the CPU 502 of Fig. 5. The device control program 901 is stored in the external storage device 505 as described above, and upon startup, the CPU 502 loads and executes the device control program 901 on the RAM 503. The device control program 901 is made up of a data transmission / reception unit 902, a data management unit 903, a scan unit 904, a print unit 905, a display unit 906, a voice operation determination unit 907, and a device control unit 908.

[0067] The data transmission / reception unit 902 transmits and receives data by TCP / IP with other devices on the network 104 via the network I / F 506. For example, the data transmission / reception unit 902 receives an operation instruction command and a filter display command from the server 102. The data transmission / reception unit 902 also transmits to the server 102 a screen update notification indicating that the screen display content of the touch panel 200 has been updated, and a job execution status notification indicating the status of a job.

[0068] The data management unit 903 stores and manages various data, such as work data generated in the execution of the device control program 901 and setting parameters required for controlling each device, in a predetermined area on the RAM 503 and the external storage device 505. For example, job data consisting of a combination of each setting item and setting value of a job executed by the device control unit 908, and machine setting information in which attribute information of paper and the like are set are managed. In addition, authentication information required for communication with the gateway 105, device information, URL, authentication information, and the like required for communication with the server 102 are managed. In addition, image data to be formed by the image forming apparatus 101 is stored and managed. In addition, screen control information used by the display unit 906 for screen display control and voice operation determination information used by the voice operation determination unit 907 for determining an operation are stored, and the screen control information and voice operation determination information are managed for each screen displayed by the display unit 906. In addition, commands and control means for voice recognition activation and voice operation activation by the network I / F 506 or other activation means, and the like are managed.

[0069] The scanning unit 904 executes scanning with a scanner 515 via a scan controller 514 based on the scan job parameter settings of the device control unit 908, and stores the read image data in a data management unit 903. The printing unit 905 executes printing with a print engine 513 via a print controller 512 based on the print job parameter settings of the device control unit 908.

[0070] The display unit 906 controls the operation panel 108 via the display controller 507. The display unit 906 displays user-operable UI parts (such as buttons, pull-down lists, and check boxes) on the touch panel 200 based on the above screen control information. The display unit 906 also acquires coordinates indicating a touched position on the touch panel 200 via the operation I / F 508, and determines the UI part to be operated and the processing content at the time of receiving the operation. In response to the determination of the processing content, the display unit 906 updates the content of the screen displayed on the touch panel 200, and transmits the parameters of the job set by the user operation and a start instruction for the job to the device control unit 908. In addition, when a voice operation is started, the display unit 906 displays voice recognition information in association with the UI parts, as shown in FIG. 11(a) to be described later. Then, in response to a voice operation determination result of the voice operation determination unit 907, the display unit 906 updates the content of the screen displayed on the touch panel 200, and transmits the parameters of the job set by the user operation and a start instruction for the job to the device control unit 908.

[0071] The voice operation determination unit 907 determines, as an operation target, a user-operable UI part constituting a screen displayed on the touch panel 200, based on an operation instruction command received by the data transmission / reception unit 902 from the server 102. For example, when the home screen 201 is displayed and an operation instruction command of "start copy" is received, the image forming apparatus 101 displays a copy screen 1112 shown in FIG. 11(d) to be described later on the touch panel 200. When an operation instruction command of "start" is received in this state, the image forming apparatus 101 executes copying. In this way, after the status during voice recognition is displayed, the user 106 utters "copy" and "start" to the voice control device 100, and the image forming apparatus 101 starts copying in the default setting state of the copy screen.

[0072] The device control unit 908 issues control instructions to the print engine 513 and the scanner 515 via the print controller 512 and the scan controller 514. For example, when detecting pressing of the start key on the touch panel 200 while a copy screen 1112 of Fig. 11(d) described later is displayed on the touch panel 200, the device control unit 908 receives copy job parameters and a job start instruction from the display unit 906. Based on the job parameters, the device control unit 908 controls the print engine 513 to print image data read by the scanner 515 on a sheet.

[0073] FIG. 10 is a sequence diagram showing the procedure of the voice operation control process executed by the information processing system of the present embodiment. In this embodiment, the CPU 302 of the voice control device 100 loads the voice control program 601 stored in the external storage device 305 on the RAM 303 and executes it. In addition, the CPU 402a of the server 102 loads the voice recognition program 701 stored in the external storage device 405a on the RAM 403a and executes it, and the CPU 402b loads the remote control program 801 stored in the external storage device 405b on the RAM 403b and executes it. Furthermore, the CPU 502 of the image forming apparatus 101 loads the device control program 901 stored in the external storage device 505 on the RAM 503 and executes it. In this way, the voice operation control process of FIG. 10 is realized. In FIG. 10, as an example, a case will be described where the user 106 utters "Start voice operation", "Copy", and "Copy". In addition, in the display control process of FIG. 10, it is assumed that the image forming apparatus 101 is started and the home screen 201 is displayed on the touch panel 200.

[0074] 10, first, the user 106 speaks "start voice operation" to the microphone 308 of the voice control device 100 (step S1001). The voice control program 601 stores "start voice operation" spoken by the user 106 as voice data in the external storage device 305. Next, the voice control program 601 transmits the voice data of "start voice operation" to the server 102 (step S1002).

[0075] The voice recognition program 701 of the server 102 performs voice recognition processing based on the received voice data (step S1003). Specifically, the voice recognition program 701 stores the received voice data in the external storage device 405a, and converts the voice data into text data by the voice recognition unit 705. Next, the voice recognition program 701 transmits the converted text data to the remote control program 801 (step S1004).

[0076] The remote control program 801 transmits to the image forming apparatus 101 an operation instruction command corresponding to the received text data, specifically, "start voice operation," from among the multiple operation instruction commands managed by the data management unit 803 (step S1005).

[0077] When the device control program 901 of the image forming apparatus 101 receives the operation instruction command ("start voice operation"), it displays a message indicating that voice operation is possible, specifically, "voice recognition in progress", on the status display 210 of the home screen 201 (step S1006). Next, the device control program 901 transmits to the server 102 screen information indicating the type of screen displayed on the touch panel 200 and UI widget information indicating UI widgets included in the screen (step S1007). Specifically, the device control program 901 transmits "home screen 201" indicating the home screen 201 displayed on the touch panel 200 to the server 102 as screen information.

[0078] The remote control program 801 of the server 102 identifies UI widget information corresponding to the home screen 201 from among a plurality of pieces of UI widget information managed by the data management unit 803, based on the screen information received from the image forming apparatus 101. The remote control program 801 also identifies voice recognition information corresponding to the identified UI widget information from among a plurality of pieces of voice recognition information managed by the data management unit 803. The remote control program 801 transmits all of the identified UI widget information and all of the identified voice recognition information to the image forming apparatus 101 (step S1008).

[0079] The device control program 901 of the image forming apparatus 101 displays the voice recognition information as an example of speech in association with each UI component displayed on the home screen 201 based on the received UI component information and voice recognition information (step S1009). Specifically, the device control program 901 displays a speech bubble 1101 in FIG. 11A including the voice recognition information of the icon “Copy” in association with a UI component that can be operated by voice, for example, the copy 202. The device control program 901 also displays information 1102 in FIG. 11A indicating that voice operation is not possible in association with a UI component that cannot be operated by voice, for example, the secure print 206. The device control program 901 also displays a message such as “Use the displayed word for voice operation” in the status display 210, and displays a speech bubble 1103 in FIG. 11A including “Delete word” in association with the status display 210. “Delete word” is voice recognition information for switching the voice recognition information displayed on the screen of the touch panel 200 to hidden. Next, the device control program 901 transmits to the server 102 a screen update notification indicating that the display content on the screen of the touch panel 200 has been updated (step S1010).

[0080] Upon receiving the screen update notification, the remote control program 801 of the server 102 transmits response text data indicating a response message to the content uttered by the user 106 in step S1001 to the voice recognition program 701 (step S1011). Specifically, the remote control program 801 transmits response text data including a message such as "You can operate with the displayed word" to the voice recognition program 701. The voice recognition program 701 converts the received response text data into voice synthesis data, and transmits the voice synthesis data to the voice control device 100 (step S1012).

[0081] The voice control program 601 of the voice control device 100 outputs the received voice synthesis data from the speaker 310 (voice output control means). Specifically, the voice control program 601 outputs a voice message such as "You can operate it with the displayed word" from the speaker 310 (step S1013). Thus, in this embodiment, the image forming device 101 can be voice-operated by the user 106 saying "Start voice operation" into the microphone 308.

[0082] Thereafter, when the user 106 speaks "copy" into the microphone 308 of the voice control device 100 (step S1014), the voice control program 601 stores "copy" spoken by the user 106 as voice data in the external storage device 305. Next, the voice control program 601 transmits the voice data of "copy" to the server 102 (step S1015).

[0083] The voice recognition program 701 of the server 102 performs voice recognition processing based on the received voice data, similarly to step S1003 described above (step S1016). Next, the voice recognition program 701 transmits the converted text data to the remote control program 801 (step S1017). Next, the remote control program 801 compares the received text data with information managed by the data management unit 803. Specifically, the remote control program 801 compares the received text data with a plurality of pieces of voice recognition information managed by the data management unit 803. For example, the plurality of pieces of voice recognition information in FIG. 8B managed by the data management unit 803 does not include voice recognition information that matches the received text data, "copy". In this case, the remote control program 801 compares the received text data with a plurality of filter words managed by the data management unit 803. For example, the plurality of filter words in FIG. 8B managed by the data management unit 803 includes a filter word that matches "copy". In this case, the remote control program 801 identifies UI widget information whose corresponding filter word is "copy" from among a plurality of pieces of UI widget information managed by the data management unit 803. The remote control program 801 also identifies voice recognition information corresponding to the identified UI widget information from among a plurality of pieces of voice recognition information managed by the data management unit 803. The remote control program 801 transmits a filter display command including all of the identified UI widget information and all of the identified voice recognition information to the image forming apparatus 101 (step S1018).

[0084] The device control program 901 of the image forming apparatus 101 performs a filter display of UI parts related to "copy" based on the received filter display command (step S1019). As a result, a home screen 1104 of FIG. 11(b) is displayed on the touch panel 200. The home screen 1104 includes icons related to "copy", such as copy 202, offset prevention copy 1105, economical copy 1106, preset ID card copy 1107, dark text copy 1108, passport copy 1109, and ID card copy 1110. Also, similar to the above-mentioned step S1009, voice recognition information is additionally displayed as an example of speech in association with these icons. Furthermore, a message such as "Copy-related icons have been filtered and displayed" is displayed on the status display 210. Note that in step S1019, the device control program 901 may display a list of functions related to "copy" together, as in the speech bubble 1111 of FIG. 11(c). By displaying the functions related to "copy" in a list in this manner, the user can easily find the desired function, and the operability by voice operation can be improved. In addition, in this embodiment, when the icons related to "copy" cannot be displayed on the home screen, the icons related to "copy" may be displayed together in a pop-up screen by scrolling. In this embodiment, when the number of icons related to "copy" is greater than a predetermined number set in advance, the voice control device 100 may be controlled to output a voice message such as "There are too many matching items". Next, the device control program 901 transmits a screen update notification indicating that the display contents of the screen of the touch panel 200 have been updated to the server 102 (step S1020). The screen update notification includes screen information indicating the type of the screen displayed on the touch panel 200 in step S1019.

[0085] Upon receiving the screen update notification, the remote control program 801 of the server 102 stores the screen information included in the screen update notification in the external storage device 405b. The remote control program 801 also transmits response text data including a message such as "Filtered by copying" to the voice recognition program 701 (step S1021). The voice recognition program 701 converts the received response text data into voice synthesis data, and transmits the voice synthesis data to the voice control device 100 (step S1022).

[0086] The voice control program 601 of the voice control device 100 outputs the received voice synthesis data from the speaker 310. Specifically, the voice control program 601 outputs a voice message such as "Filtered with copy" from the speaker 310 (step S1023). Thus, in this embodiment, when the user 106 speaks "copy" into the microphone 308, a filtered display of UI components related to "copy" is performed on the touch panel 200 of the image forming device 101.

[0087] After that, the user 106 speaks "Copy it" to the microphone 308 of the voice control device 100 (step S1024). The voice control program 601 stores "Copy it" spoken by the user 106 as voice data in the external storage device 305. Next, the voice control program 601 transmits the voice data of "Copy it" to the server 102 (step S1025).

[0088] The voice recognition program 701 of the server 102 performs voice recognition processing based on the received voice data (step S1026), similar to step S1003 described above, and transmits the converted text data to the remote control program 801 (step S1027).

[0089] The remote control program 801 transmits an operation instruction command corresponding to the received text data, specifically, "start copying," from among a plurality of operation instruction commands managed by the data management unit 803, to the image forming apparatus 101 (step S1028).

[0090] When the device control program 901 of the image forming apparatus 101 receives the operation instruction command ("start copy"), it displays a copy screen for executing a copy job on the touch panel 200 (step S1029). At this point, voice recognition information such as that shown in FIG. 11(a) is not displayed on the copy screen. Next, the device control program 901 transmits screen information indicating the type of screen displayed on the touch panel 200 and UI widget information indicating UI widgets included in the screen to the server 102 (step S1030).

[0091] Based on the received screen information, the remote control program 801 of the server 102 identifies UI widget information corresponding to the copy screen from among a plurality of pieces of UI widget information managed by the data management unit 803. The remote control program 801 also identifies voice recognition information corresponding to the identified UI widget information from among a plurality of pieces of voice recognition information managed by the data management unit 803. The remote control program 801 transmits all of the identified UI widget information and all of the identified voice recognition information to the image forming apparatus 101 (step S1031).

[0092] Based on the received voice recognition information, the device control program 901 of the image forming apparatus 101 displays the voice recognition information as an example of speech in association with each UI component displayed on the copy screen (step S1032). In step S1032, as in the copy screen 1112 of Fig. 11(d), a speech bubble containing the corresponding voice recognition information is displayed as an example of speech in association with each UI component included in the copy screen 1112. Next, the device control program 901 transmits a screen update notification including screen information indicating the type of screen displayed on the touch panel 200 to the server 102 (step S1033).

[0093] Upon receiving the screen update notification, the remote control program 801 of the server 102 transmits response text data indicating a response message to the content uttered by the user 106 in step S1024 to the voice recognition program 701. Specifically, the remote control program 801 transmits response text data including a message such as "Copy screen has been displayed" to the voice recognition program 701 (step S1034). The voice recognition program 701 converts the received response text data into voice synthesis data, and transmits the voice synthesis data to the voice control device 100 (step S1035).

[0094] The voice control program 601 of the voice control device 100 outputs the received voice synthesis data from the speaker 310. Specifically, the voice control program 601 outputs a voice message such as "Copy screen has been displayed" from the speaker 310 (step S1036).

[0095] As described above, in this embodiment, the image forming apparatus 101 is operated by voice based on the voice of the user 106 acquired by the voice control apparatus 100.

[0096] In the above-described process shown in FIG. 10, the voice recognition information is displayed in association with the UI widget. However, it is also possible to hide the displayed voice recognition information.

[0097] For example, as shown in FIG. 11(a), when voice recognition information is displayed in correspondence with a UI component included in the home screen 201 and the user 106 speaks "delete the word" into the microphone 308, the voice control device 100 transmits "delete the word" as voice data to the server 102.

[0098] The server 102 performs voice recognition processing based on the received voice data to generate text data, and transmits to the image forming device 101 an operation instruction command corresponding to the above text data, specifically, "delete word," from among multiple operation instruction commands managed by the data management unit 803.

[0099] When the image forming apparatus 101 receives the operation instruction command ("delete word"), it hides the voice recognition information on the home screen 201. As a result, the home screen 201 (see FIG. 2, for example) that does not include the voice recognition information is displayed on the touch panel 200. Next, the image forming apparatus 101 transmits a screen update notification including screen information indicating the type of the screen displayed on the touch panel 200 to the server 102.

[0100] The server 102 that has received the screen update notification converts response text data indicating a response message to the content uttered by the user 106 into voice synthesis data, and transmits the voice synthesis data to the voice control device 100 .

[0101] The voice control device 100 outputs the received voice synthesis data from the speaker 310. Specifically, the voice control program 601 outputs from the speaker 310 a voice message such as "The word has been deleted. Voice control is possible."

[0102] In this manner, in this embodiment, it is possible to easily switch between displaying and not displaying the voice recognition information based on the voice spoken by the user.

[0103] Next, the operations of the image forming apparatus 101, the server 102, and the voice control apparatus 100 for implementing voice operations of the information processing system in this embodiment will be described.

[0104] Fig. 12 is a flowchart showing the procedure of the screen update control process executed by the device control program 901 of Fig. 9. The screen update control process of Fig. 12 is realized by the CPU 502 of the image forming apparatus 101 expanding the device control program 901 stored in the external storage device 505 onto the RAM 503 and executing it. In the screen update control process of Fig. 12, it is assumed that the image forming apparatus 101 has started up and the home screen 201 is displayed on the touch panel 200.

[0105] In FIG. 12, first, when the device control program 901 receives voice operation information such as the above-mentioned operation instruction command or filter display command from the server 102 (step S1201), the device control program 901 determines whether the voice operation information is an operation instruction command (step S1202).

[0106] If it is determined in step S1202 that the received voice operation information is an operation instruction command, the device control program 901 performs display control of the touch panel corresponding to the operation instruction command (step S1203). For example, if the operation instruction command is "start voice operation", the device control program 901 displays "voice recognition in progress" in the status display 210 of the home screen 201 (see, for example, step S1006 described above). If the operation instruction command is "display menu", the device control program 901 displays a menu screen (not shown) on the touch panel 200. If the operation instruction command is an instruction to execute a job, the device control program 901 executes the job in parallel. For example, if the operation instruction command is an instruction to operate the start button on the copy screen 1112, the copy job is executed and a copy execution screen (not shown) indicating that the copy job is being executed is displayed on the touch panel 200.

[0107] Next, the device control program 901 transmits to the server 102 screen information indicating the type of screen displayed on the touch panel 200 and UI widget information indicating UI widgets included in the screen (step S1204). The server 102, which has received the screen information from the image forming apparatus 101, identifies UI widget information corresponding to the screen information from among the multiple pieces of UI widget information managed by the data management unit 803. The server 102 also identifies voice recognition information corresponding to the identified UI widget information from among the multiple pieces of voice recognition information managed by the data management unit 803. The remote control program 801 transmits all of the identified UI widget information and all of the identified voice recognition information to the image forming apparatus 101.

[0108] Next, the device control program 901 receives the UI widget information and the voice recognition information from the server 102 (step S1205). Next, the device control program 901 displays the voice recognition information in association with each UI widget displayed on the screen of the touch panel 200 (step S1206) (see, for example, steps S1009 and S1032 described above). Here, for example, if a UI widget corresponding to the voice recognition information received from the server 102 is not displayed on the screen of the touch panel 200, the device control program 901 may display a list of all the voice recognition information for which no corresponding UI widget exists, as in the speech bubble 1111 in FIG. 11(c).

[0109] Next, the device control program 901 transmits a screen update notification including screen information indicating the type of screen displayed on the touch panel 200 to the server 102 (step S1207), and ends this process.

[0110] If it is determined in step S1202 that the received voice operation information is not an operation instruction command, the device control program 901 determines whether the received voice operation information is a filter display command (step S1208).

[0111] If it is determined in step S1208 that the received voice operation information is a filter display command, the device control program 901 performs filter display based on the received filter display command (step S1209) (see, for example, step S1019 described above). Through the process of step S1209, a screen including UI widgets filtered by the words spoken by the user 106 into the microphone 308 is displayed on the touch panel 200 based on the UI widget information included in the received filter display command. Furthermore, the voice recognition information included in the received filter display command is displayed in association with each UI widget. Thereafter, the screen update control process proceeds to step S1207. At this time, the screen update notification transmitted in step S1207 may include screen information indicating the screen that was displayed on the touch panel 200 immediately before the filter display was performed, or may include screen information indicating a new screen different from the screen.

[0112] If it is determined in step S1208 that the received voice operation information is not a filter display command, the screen update control process proceeds to step S1210. If the received voice operation information is not a filter display command, for example, the voice operation information is information indicating an operation instruction for the secure print 206 for which voice operation is not permitted. In step S1210, the device control program 901 transmits a voice instruction failure response to the server 102 indicating that an unexecutable command has been received. Thereafter, the screen update control process ends. After ending the screen update control process, the device control program 901 enters a state of waiting to receive voice operation information from the server 102.

[0113] Fig. 13 is a flowchart showing the procedure of the voice control process executed by the voice control program 601 of Fig. 6. The voice control process of Fig. 13 is realized by the CPU 302 of the voice control device 100 expanding the voice control program 601 stored in the external storage device 305 on the RAM 303 and executing it.

[0114] In FIG. 13, the voice control program 601 acquires the voice spoken by the user to the microphone 308 by the voice acquisition unit 604 (step S1301). The voice control program 601 determines the end of the acquired voice by the speech end determination unit 608, converts the voice from the start of the speech to the end of the speech into voice data, and stores the voice data in the external storage device 305. Next, the voice control program 601 determines whether or not voice data has been detected (step S1302). In step S1302, for example, when the process of storing the voice data in the external storage device 305 is completed, the voice control program 601 determines that the voice data has been detected. On the other hand, when the end of the acquired voice is not determined by the speech end determination unit 608 and the process of storing the voice data in the external storage device 305 is not completed, the voice control program 601 determines that the voice data has not been detected.

[0115] If it is determined in step S1302 that no voice data is detected, the voice control process returns to step S1301. If it is determined in step S1302 that voice data is detected, the voice control program 601 causes the display unit 606 to blink the LED 312 (step S1303) to notify that the voice control device 100 is in a response processing state. Next, the voice control program 601 transmits the voice data stored in the external storage device 305 to the server 102 via the data transmission / reception unit 602 (step S1304). Thereafter, the voice control program 601 waits until it receives voice synthesis data from the server 102.

[0116] When the voice synthesis data is received from the server 102 (YES in step S1305), the voice control program 601 causes the voice playback unit 605 to play back the received voice synthesis data (step S1306). Next, the voice control program 601 determines whether or not an interaction session end notification has been received from the server 102 (step S1307).

[0117] If it is determined in step S1307 that an interaction session end notification has not been received from the server 102, the voice control process returns to step S1301. If it is determined in step S1307 that an interaction session end notification has been received from the server 102, the voice control program 601 causes the display unit 606 to turn off the LED 312 (step S1308) and notifies that the voice control device 100 is in a standby state. Next, the voice control program 601 ends the interaction session (step S1309), and the voice control process ends. After the voice control process ends, when the user speaks into the microphone 308, the process of step S1301 is executed.

[0118] Fig. 14 is a flowchart showing the procedure of the voice recognition control process executed by the voice recognition program 701 of Fig. 7. The voice recognition control process of Fig. 14 is realized by the CPU 402a of the server 102 expanding the voice recognition program 701 stored in the external storage device 405a on the RAM 403a and executing it.

[0119] 14, the voice recognition program 701 receives voice data or text data through the data transmission / reception unit 702 (step S1401). The voice recognition program 701 stores the received data and information on the sender of the data in the external storage device 405a through the data management unit 703. Next, the voice recognition program 701 determines whether the data stored in the external storage device 405a is voice data (step S1402).

[0120] If it is determined in step S1402 that the data stored in the external storage device 405a is voice data, the voice recognition program 701 causes the voice recognition unit 705 to perform voice recognition processing on the received voice data (step S1403). The voice recognition program 701 also causes the data management unit 703 to store the results of the voice recognition processing in the external storage device 405a. Next, the voice recognition program 701 causes the morphological analysis unit 706 to perform morphological analysis on the stored results of the voice recognition processing to generate text data. The voice recognition program 701 transmits this text data to the remote control program 801 (step S1404), and the voice recognition control processing ends.

[0121] If it is determined in step S1402 that the data stored in the external storage device 405a is not voice data, the voice recognition program 701 determines whether the data stored in the external storage device 405a is response text data (step S1405).

[0122] If it is determined in step S1405 that the data stored in the external storage device 405a is response text data, the voice recognition program 701 generates voice synthesis data by performing voice synthesis processing on the data stored in the external storage device 405a using the voice synthesis unit 707 (step S1406). Next, the voice recognition program 701 transmits the generated voice synthesis data to the voice control device 100 (step S1407), and the voice recognition control process ends.

[0123] If the result of the determination in step S1405 is that the data stored in the external storage device 405a is not response text data, the voice recognition program 701 determines that the data stored in the external storage device 405a is invalid data. The voice recognition program 701 generates voice synthesis data for reproducing an error message such as "Voice not recognized" (step S1408). After that, the voice recognition control process proceeds to step S1407.

[0124] Fig. 15 is a flowchart showing the procedure of the remote control process executed by the remote control program 801 in Fig. 8A. The remote control process in Fig. 15 is realized by the CPU 402b of the server 102 expanding the remote control program 801 stored in the external storage device 405b on the RAM 403b and executing it.

[0125] 15, the remote control program 801 receives text data from the voice recognition program 701 through the data transmission / reception unit 802 (step S1501). The remote control program 801 stores this text data in the external storage device 405b through the data management unit 803. Next, the remote control program 801 compares the received text data with a plurality of pieces of voice recognition information managed by the data management unit 803. The remote control program 801 determines whether or not the plurality of pieces of voice recognition information managed by the data management unit 803 includes voice recognition information that matches the received text data (step S1502).

[0126] If it is determined in step S1502 that the voice recognition information managed by data management unit 803 includes voice recognition information that matches the received text data, remote control program 801 transmits an operation instruction command corresponding to the voice recognition information to image forming apparatus 101 via data transmission / reception unit 802 (step S1503). Having received the operation instruction command, image forming apparatus 101 performs the process of step S1203 described above to control the display of the touch panel corresponding to the received operation instruction command. In addition, image forming apparatus 101 performs the process of step S1204 described above to transmit screen information indicating the type of screen displayed on touch panel 200 and UI widget information indicating UI widgets included in the screen to server 102.

[0127] The remote control program 801 receives screen information and UI widget information from the image forming apparatus 101 via the data transmission / reception unit 802 (step S1504). Next, the remote control program 801 identifies voice recognition information corresponding to the received UI widget information from among multiple pieces of voice recognition information managed by the data management unit 803. The remote control program 801 transmits all of the identified voice recognition information to the image forming apparatus 101 via the data transmission / reception unit 802 (step S1505). Next, the remote control process proceeds to step S1508, which will be described later.

[0128] If the result of the determination in step S1502 is that the multiple pieces of voice recognition information managed by data management unit 803 do not contain voice recognition information that matches the received text data, remote control program 801 compares the received text data with multiple filter words managed by data management unit 803. Remote control program 801 determines whether the multiple filter words managed by data management unit 803 contain a filter word that matches the received text data (step S1506).

[0129] If it is determined in step S1506 that the filter words managed by data management unit 803 include a filter word that matches the received text data, remote control program 801 transmits a filter display command to image forming apparatus 101 (step S1507). As described above, this filter display command includes voice recognition information whose corresponding filter word matches the received text data, and UI widget information corresponding to the voice recognition information.

[0130] Next, when the remote control program 801 receives a response to the voice operation information, such as an operation instruction command or a filter display command, from the image forming apparatus 101 (step S1508), it determines whether the received response is a screen update notification (step S1509).

[0131] If it is determined in step S1509 that the received response is a screen update notification, the remote control program 801 stores the screen information included in the received screen update notification in the external storage device 405b via the data management unit 803 (step S1510). Next, the remote control program 801 transmits the response text data to the voice recognition program 701 (step S1511), and the remote control process ends.

[0132] If it is determined in step S1509 that the received response is not a screen update notification, the remote control program 801 determines whether the received response is a voice instruction failure response indicating that an unexecutable command has been received (step S1512).

[0133] If it is determined in step S1512 that the received response is a voice instruction failure response, the remote control program 801 transmits response text data including a message indicating that a voice operation could not be performed in response to the received voice operation information, such as "Voice operation not possible," to the voice recognition program 701 (step S1513). Then, the remote control process ends.

[0134] If it is determined in step S1512 that the received response is not the voice instruction failure response, or if it is determined in step S1506 that the filter words managed by the data management unit 803 do not include a filter word that matches the received text data, the remote control program 801 transmits an invalid data response indicating that the voice operation for the text data received in step S1501 is invalid to the voice recognition program 701 (step S1514).Then, the remote control process ends.

[0135] According to the above-described embodiment, text data is output based on voice data obtained by encoding a voice acquired by the voice control device 100, and a predetermined process associated with a UI widget is executed based on the voice recognition information and the text data. The voice recognition information includes at least words related to the predetermined process associated with the UI widget, and is displayed in association with the UI widget included in the screen being displayed on the touch panel 200 of the image forming device 101. In other words, the voice recognition information having a semantic association with the process executed by the function corresponding to the UI widget is displayed in association with the UI widget. This allows the user to easily become familiar with the association between the process to be executed and the spoken instruction.

[0136] Furthermore, in the above-described embodiment, if the text data does not match the character recognition information, a filter is performed to display UI widgets related to the text data, and voice recognition information is displayed in association with the UI widgets. This makes it easier to find a UI widget corresponding to a desired function by narrowing down the displayed UI widgets to those related to the text data, while maintaining ease of learning the connection between the processing to be executed and the spoken instruction based on the voice recognition information displayed in association with the UI widgets.

[0137] In the above-described embodiment, the UI components are icons, marks, buttons, arrows, or tabs, so that the user can easily become familiar with the association between each process corresponding to the icons, marks, buttons, arrows, and tabs displayed on the touch panel 200 and speech instructions.

[0138] In the above-described embodiment, the speech recognition information is displayed in a speech bubble (see, for example, speech bubble 1101) that is displayed in association with the UI widget. This allows the user to easily understand the correspondence between the UI widget and the speech recognition information.

[0139] In the embodiment described above, information 1102 indicating that voice operation is not permitted is displayed in association with the secure print 206 corresponding to a function for which voice operation is not permitted among UI widgets included in the screen being displayed on the touch panel 200. This allows the user to easily understand that the secure print 206 cannot be operated by voice.

[0140] In the above-described embodiment, the predetermined process associated with the UI widget is a print process, a process for accepting settings related to the print process, a read process, or a process for accepting settings related to the read process. This allows the user to easily accept settings for the print process or the read process or to perform voice operations to instruct execution.

[0141] Although the present invention has been described above using the above-mentioned embodiment, the present invention is not limited to the above-mentioned embodiment. For example, the data management unit 803 may not manage filter words, and may specify UI parts to be displayed in a filtered manner from words related to the voice recognition information (synonyms, similar words, web search results).

[0142] Also, in the above-described embodiment, the voice recognition program 701 and the remote control program 801 may be stored in the external storage device 505 of the image forming apparatus 101, and the image forming apparatus 101 may execute the voice recognition control process of Fig. 14 and the remote control process of Fig. 15. In such a configuration, the image forming apparatus 101 acquires voice data obtained by encoding the voice of the user 106 from the voice control device 100 or a microphone provided in the image forming apparatus 101. In this manner, in a configuration in which the image forming apparatus 101 performs voice recognition without going through the server 102, the user can easily become familiar with the association between the process to be executed and spoken instructions.

[0143] In the above-described embodiment, the home screen of the image forming apparatus 101 is not limited to the home screen 201, and similar control is performed for other home screens.

[0144] Fig. 16 is a diagram showing an example of a home screen 201b displayed on the touch panel 200 of Fig. 2. The home screen 202b is a home screen corresponding to tab 3. The home screen 202b includes icons related to the scan function of the image forming apparatus 101, specifically, Frequently Used Settings A 1601, Frequently Used Settings B 1602, Send with Fixed Text 1603, Fixed Destination Scan 1604, and Assortment Scan 1605.

[0145] Frequently used setting A 1601 is an icon for instructing the execution of a process of scanning a document based on a scan setting registered in advance, generating image data, and transmitting the image data. Frequently used setting B 1602 is an icon for instructing the execution of a process of scanning a document based on a scan setting different from the frequently used setting A 1601, generating image data, and transmitting the image data. Send with fixed text 1603 is an icon for instructing the execution of a process of scanning a document, generating image data, attaching the image data to text information in the body of an e-mail registered in advance, and sending the e-mail. The text information in the body of an e-mail registered in advance is, for example, a fixed text such as "Documents are attached. Thank you for your cooperation."

[0146] Fixed destination scan 1604 is an icon for issuing an instruction to execute a process of scanning an original document, generating image data, and sending the image data to a preregistered destination Z. Sorting scan 1605 is an icon for issuing an instruction to execute a process of scanning an original document, generating image data, and saving the name of the image data as the date and time of scanning, etc.

[0147] When the home screen 201b is displayed on the touch panel 200 and the user speaks "Start voice operation" into the microphone 308 of the voice control device 100, character recognition information is displayed in association with each icon on the home screen 201b, as shown in FIG. 16.

[0148] In the above-described embodiment, the voice recognition information and filter words may be set by utilizing the Web server function of the remote control program 801 .

[0149] FIG. 17 is a diagram showing an example of a setting screen 1700 for setting voice recognition information and filter words in the present embodiment. FIG. 17 shows a state in which a home screen 201 corresponding to tab 1, a home screen 201a corresponding to tab 2, a home screen 201b corresponding to tab 3, and text boxes corresponding to them are displayed. The setting screen 1700 is displayed by accessing a remote control program 801 of the server 102 with a Web browser. The name of an icon included in the corresponding home screen is set in the "position" column of the text box. The user can set voice recognition information and filter words in association with the name of each icon. Note that in each setting field of voice recognition information and filter words corresponding to the name of an icon for which voice operation is not permitted, such as the secure print 206, "-" indicating that voice recognition information and filter words cannot be set is input in advance. In setting filter words, multiple filter words can be set by inserting a delimiter "," between words. In addition, in the setting screen 1700, it is also possible to change screen information and UI widget information.

[0150] Fig. 18 is a flowchart showing the procedure of the setting control process executed by the remote control program 801 of Fig. 8A. The setting control process of Fig. 18 is realized by the CPU 402b of the server 102 expanding the remote control program 801 stored in the external storage device 405b on the RAM 403b and executing it.

[0151] 18, the remote control program 801 receives data through the data transmission / reception unit 802 (step S1801). Next, the remote control program 801 determines whether the received data is text data transmitted from the voice recognition program 701 (step S1802).

[0152] If it is determined in step S1802 that the received data is text data sent from the voice recognition program 701, the remote control program 801 executes the remote control process of FIG. 15 (step S1803), and the setting control process ends.

[0153] If the result of the determination in step S1802 is that the received data is not text data sent from the voice recognition program 701, the remote control program 801 determines whether the received data is an access notification sent from a Web browser such as the client terminal 103 (step S1804).

[0154] If it is determined in step S1804 that the received data is not an access notification sent from the Web browser of the client terminal 103, the remote control program 801 sends an invalid response indicating that the data cannot be processed to the sender of the received data (step S1805). Then, the setting control process ends.

[0155] If it is determined in step S1804 that the received data is an access notification transmitted from the Web browser of the client terminal 103, the remote control program 801 acquires information managed by the data management unit 803 (step S1806). Specifically, the remote control program 801 acquires screen information, UI widget information, voice recognition information, and filter word information managed by the data management unit 803. Next, the remote control program 801 generates setting screen display data for displaying the setting screen 1700 on the Web browser of the client terminal 103 based on the information acquired in step S1806 (step S1807). Next, the remote control program 801 transmits the generated setting screen display data to the client terminal 103 (step S1808). The client terminal 103 displays the setting screen 1700 on the Web browser of the client terminal 103 based on the received setting screen display data. The client terminal 103 transmits the setting data set by the user on the setting screen 1700 to the server 102. The setting data includes, for example, voice recognition information and filter words changed by the user so that the corresponding screen information and UI widget information can be identified. The setting data also includes screen information of a new screen set by the user and UI widget information corresponding to the screen information.

[0156] Next, the remote control program 801 receives the setting data from the client terminal 103 (step S1809). Next, the remote control program 801 identifies the changed data (hereinafter referred to as "change data") based on the setting data and the information acquired in step S1806, and acquires the identified change data (step S1810). In step S1810, if multiple pieces of change data are identified, the remote control program 801 acquires one piece of change data from the multiple pieces of change data. Next, the remote control program 801 determines whether the acquired change data is voice recognition information (step S1811).

[0157] If it is determined in step S1811 that the acquired change data is voice recognition information, the remote control program 801 identifies voice recognition information corresponding to the same screen information and UI widget information as the change data from among the multiple pieces of voice recognition information managed by the data management unit 803. The remote control program 801 changes the identified voice recognition information to the change data (step S1812). Next, the setting control process proceeds to step S1817, which will be described later.

[0158] If it is determined in step S1811 that the acquired change data is not voice recognition information, the remote control program 801 determines whether the acquired change data is a filter word (step S1813).

[0159] If it is determined in step S1813 that the acquired change data is a filter word, the remote control program 801 identifies a filter word that corresponds to the same screen information and UI widget information as the change data from among a plurality of filter words managed by the data management unit 803. The remote control program 801 changes the identified filter word to the change data (step S1814). Next, the setting control process proceeds to step S1817, which will be described later.

[0160] If it is determined in step S1813 that the acquired change data is not a filter word, the remote control program 801 determines whether the acquired change data is screen information or UI widget information (step S1815).

[0161] If it is determined in step S1815 that the acquired change data is neither screen information nor UI widget information, the setting control process ends. If it is determined in step S1815 that the acquired change data is screen information or UI widget information, the remote control program 801 identifies information corresponding to the change data from among a plurality of pieces of screen information and UI widget information managed by the data management unit 803. The remote control program 801 changes the identified information to the change data (step S1816). Next, the remote control program 801 determines whether or not the processing of all the change data has been completed (step S1817).

[0162] If it is determined in step S1817 that the processing of any of the changed data has not been completed, the setting control process returns to step S1810. If it is determined in step S1817 that the processing of all of the changed data has been completed, the remote control program 801 transmits the changed information to the image forming apparatus 101 (step S1818). The image forming apparatus 101 changes the display of the home screen based on the received information. Thereafter, the setting control process ends.

[0163] By carrying out the setting control process of FIG. 18 described above, it becomes possible to easily customize the voice recognition information and filter words.

[0164] In the above embodiment, the data management unit 803 of the remote control program 801 manages the voice recognition information and filter words, but the present invention is not limited to this configuration. For example, the data management unit 903 of the device control program 901 may manage the voice recognition information and filter words. By having the data management unit 903 manage the voice recognition information and filter words, it is possible to change the settings of the voice recognition information and filter words on the touch panel 200 of the image forming apparatus 101.

[0165] In addition, in the above-described embodiment, in a configuration in which the data management unit 903 of the device control program 901 manages voice recognition information and filter words, the device control program 901 may execute the setting control process of Figure 18 by utilizing the Web server function of the device control program 901.

[0166] The present invention can also be realized by supplying a program that realizes one or more functions of the above-mentioned embodiments to a system or device via a network or a storage medium, and having one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0167] In addition, CPU stands for Central Processing Unit. DNN stands for Deep Neural Network. GMM stands for Gaussian mixture model. HDD stands for Hard Disk Drive. HMM stands for Hidden Markov Model. ID stands for Identification. IEEE stands for Institute of Electrical and Electronics Engineers. IP stands for Internet Protocol. LAN stands for Local Area Network. LCD stands for Liquid Crystal Display. LED stands for Light Emitting Diode. MEMS stands for Micro Electro Mechanical Systems. MP3 stands for MPEG Audio Layer-3. PC stands for Personal Computer. RAM stands for Random-Access Memory. RNN stands for Recurrent Neural Networks. ROM stands for Read Only Memory. SD Card stands for Secure Digital Memory Card. SSD stands for Solid State Drive. TCP stands for Transmission Control Protocol. UI stands for User Interface. URL stands for Uniform Resource Locator. [Explanation of symbols]

[0168] 100 Voice control device 101 Image forming device 102 Server 200 Touch Panel 202 Copy 203 Scan 204 Menu 205 Address Book 206 Secure Print 207 Voice Recognition 308 Microphone 402a, 402b, 502 CPU 513 Print Engine 515 Scanner 701 Voice Recognition Program 801 Remote Control Program 901 Device Control Program 1101 Speech bubble

Claims

1. A display device capable of displaying information; A microphone capable of acquiring sound; an output means for outputting word information based on natural language voice information inputted through the microphone; a display control means for additionally displaying example utterances in association with a touch object included in a screen being displayed on the display device; an execution means for executing a predetermined process associated with the touch object, The example utterance includes at least a word constituting a process name of the predetermined process, A command for executing a predetermined process associated with each of a plurality of touch objects displayed on the display device is managed in association with word information for filtering and example utterances; the execution means executes a predetermined process for linking the output word information to a touch object managed in association with the utterance example when the output word information matches information of a combination of words included in the utterance example; When the output word information does not match the information of the word combination contained in the example utterance, the display control means displays on the display device a touch object managed in association with word information for a filter that matches the output word information, and the example utterance in correspondence with the touch object.

2. The information processing system according to claim 1 , wherein the touch object is an icon, a mark, a button, an arrow, or a tab.

3. 3. The information processing system according to claim 1, wherein the display control means displays the example utterance in a speech bubble displayed in association with the touch object.

4. The information processing system according to any one of claims 1 to 3, characterized in that the display control means displays, in association with a touch object included in a screen being displayed on the display device, information indicating that voice operation is not possible for a touch object corresponding to a function for which voice operation based on the output word information is not permitted.

5. The information processing system according to any one of claims 1 to 4, characterized in that when word information instructing to hide the speech example is output while the speech example is displayed in association with the touch object, the display control means hides the speech example displayed in association with the touch object.

6. 6. The information processing system according to claim 1, wherein the display control means changes the screen in accordance with the execution of the predetermined process.

7. 7. The information processing system according to claim 1, further comprising a voice output control means for causing a voice output device to output a voice message in accordance with the execution of the predetermined process.

8. a printing device for forming an image on the sheet; 8. The information processing system according to claim 1, wherein the predetermined process is a printing process.

9. a printing device for forming an image on the sheet; 8. The information processing system according to claim 1, wherein the predetermined process is a process of accepting settings related to a printing process.

10. Further comprising a reading device for reading the document, 10. The information processing system according to claim 1, wherein the predetermined process is a reading process.

11. Further comprising a reading device for reading the document, 10. The information processing system according to claim 1, wherein the predetermined process is a process of accepting settings related to a reading process.

12. the information processing system includes an image processing device including the display device, the display control means, and the execution means, an audio control device including the microphone, and an information processing device including the output means, The voice control device includes: a transmitting means for transmitting the natural language voice information received via the microphone to the information processing device, The image processing device includes:

12. The information processing system according to claim 1, further comprising an acquisition unit that acquires the word information from the information processing device.

13. the information processing system includes an image processing device including the display device, the microphone, the display control means, and the execution means, and an information processing device including the output means; The image processing device includes: a transmitting means for transmitting voice information in a natural language received via the microphone to the information processing device; 12. The information processing system according to claim 1, further comprising: an acquisition unit that acquires the word information from the information processing device.

14. the information processing system includes an image processing device including the display device, the output means, the display control means, and the execution means, and an audio control device including the microphone; The voice control device includes:

12. The information processing system according to claim 1, further comprising a transmitting unit for transmitting voice information in a natural language received via the microphone to the image processing device.

15. A method for controlling an information processing system having a display device capable of displaying information and a microphone capable of acquiring sound, comprising: an output step of outputting word information based on natural language voice information inputted through the microphone; a display control step of additionally displaying example utterances in association with a touch object included in a screen being displayed on the display device; and executing a predetermined process associated with the touch object, The example utterance includes at least a word constituting a process name of the predetermined process, A command for executing a predetermined process associated with each of a plurality of touch objects displayed on the display device is managed in association with word information for filtering and example utterances; In the execution step, when the output word information matches information of a combination of words included in the utterance example, a predetermined process is executed to link the output word information to a touch object managed in association with the utterance example, A control method for an information processing system, characterized in that, in the display control process, if the output word information does not match information on a word combination contained in the example utterance, a touch object managed in association with word information for a filter that matches the output word information, and the example utterance are displayed on the display device in correspondence with the touch object.

16. A program for causing a computer to execute a control method for an information processing system having a display device capable of displaying information and a microphone capable of acquiring sound, The control method for the information processing system includes: an output step of outputting word information based on natural language voice information inputted through the microphone; a display control step of additionally displaying example utterances in association with a touch object included in a screen being displayed on the display device; and executing a predetermined process associated with the touch object, The example utterance includes at least a word constituting a process name of the predetermined process, A command for executing a predetermined process associated with each of a plurality of touch objects displayed on the display device is managed in association with word information for filtering and example utterances; In the execution step, when the output word information matches information of a combination of words included in the utterance example, a predetermined process is executed to link the output word information to a touch object managed in association with the utterance example, When the output word information does not match information on a word combination contained in the example utterance, the display control process displays on the display device a touch object managed in association with word information for a filter that matches the output word information, and the example utterance is associated with the touch object.

Citation Information

Patent Citations

  • Computer control interactive display system, voice command input method, and recording medium

    JP2000215022A

  • Medical treatment support system

    JP2006221583A

  • Method and device for performing user function by using voice recognition

    JP2013143151A

  • Image processing system, image processing apparatus and program

    JP2019096978A

  • Information processing system, information processing apparatus, control method, and program

    JP2020112932A