Information processing system and information processing method

Abstract

The present invention provides an information processing system and an information processing method. The information processing system includes: a display capable of displaying information; a microphone capable of acquiring sound; and one or more controllers configured to be used as: a unit configured to control to display the following screen on the display: the screen includes multiple selection objects, one of the multiple selection objects is highlighted to indicate that the selected object in the multiple selection objects is a focus target; a unit configured to acquire first identification information and second identification information based on voice information acquired via the microphone, the first identification information corresponding to the focus target, and the second identification information related to the position update processing of the focus target; and a unit configured to control to display the screen of the focus target being updated on the display based on at least information about the current focus target, the first identification information, and the second identification information.

Description

[0001] This application is a divisional application of the invention patent application with application date of June 7, 2020, application number 202010509304.6, and invention name “Information processing system, information processing device and information processing method”. Technical Field

[0002] The present disclosure relates to an information processing system capable of operating according to voice. The information processing system includes information processing apparatuses such as printers, scanners, fax devices, multifunction peripherals having a plurality of these functions, personal computers, mobile terminals, and the like. Background Art

[0003] In recent years, with the development of voice recognition technology, devices that perform processing according to voice instructions have become popular.

[0004] Japanese Patent Publication No. 2019-67258 discloses a system configured to perform a printing process according to a printing instruction given by a user by voice. Japanese Patent Publication No. 2019-67258 illustrates a method of giving a main instruction such as a printing instruction to a printing device by voice, but does not consider a method of giving a detailed instruction such as a setting change by voice. In addition, in such a device, there are various types of setting items, and there are various screen configurations and instruction methods depending on the type. Therefore, in order to provide the system with better operability, it is desirable to allow various voice instruction methods to be used depending on the configuration of the setting screen. Summary of the invention

[0005] In view of the above, the present disclosure provides an information processing system that can be operated according to voice instructions, and the voice instructions can be given in various ways according to the structure of the setting screen. Specifically, the present disclosure provides an information processing system that can be operated according to voice instructions, and the voice instructions operate a setting screen including a slider-shaped operation object or a setting screen including multiple selection objects.

[0006] In view of the above, the present disclosure provides an image forming system, which includes: an image forming device, which is configured to form an image on a sheet; a display, which is capable of displaying information; a microphone, which is capable of acquiring sound; and one or more controllers, which are configured to serve as: a unit configured to control so as to display a screen including an operation object on the display, wherein the operation object represents parameters related to settings in forming an image by a pointing position; a unit configured to acquire first identification information corresponding to the operation object and second identification information corresponding to an update process of the pointing position based on voice information representing a single-phrase voice expression acquired via the microphone; and a unit configured to control so as to display a screen including the operation object with an updated pointing position on the display based on at least information about a current pointing position represented by the operation object, the first identification information, and the second identification information.

[0007] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a diagram illustrating a system configuration.

[0009] Figure 2 It is a diagram illustrating the structure of the operation panel.

[0010] Figure 3 This is a conceptual diagram illustrating the hardware configuration of the voice control device.

[0011] Figure 4 This is a conceptual diagram illustrating the hardware configuration of a server.

[0012] Figure 5 This is a conceptual diagram illustrating a hardware configuration of an image forming apparatus.

[0013] Figure 6 This is a conceptual diagram illustrating the functional structure of a control program of a voice control device.

[0014] Fig. 7A This is a conceptual diagram illustrating the functional structure of a control program of a server.

[0015] Figure 7B is a diagram illustrating an example of a group ID list used by the group ID determination unit to determine the group ID.

[0016] Figure 8 This is a conceptual diagram illustrating a functional structure of a control program of the image forming apparatus.

[0017] 9A to 9C This is a sequence diagram illustrating a sequence performed between devices included in the system, control programs of the devices, and users.

[0018] Fig.10 This is a diagram illustrating a process flow of a device control program of the image forming apparatus.

[0019] FIG. 11A to FIG. 11H is a diagram illustrating an example of a group ID list managed and used by a control program.

[0020] Fig. 12A is a diagram illustrating a screen displayed by a control program of an image forming apparatus, wherein the screen is from Fig.12D The Chinese version of the translation of the Japanese version shown in .

[0021] Fig. 12B FIG. 1 is a diagram illustrating an example of screen control information managed and used by a control program of the image forming apparatus.

[0022] Fig. 12C is a diagram illustrating an example of voice control operation determination information.

[0023] Fig.12D is a diagram illustrating a screen displayed by a control program of an image forming apparatus, wherein the screen is Fig. 12A The Chinese version shown in corresponds to the Japanese version.

[0024] Fig.13 2 is a diagram illustrating a flow of processing performed by a voice-controlled operation determination unit of the image forming apparatus.

[0025] Fig.14A : is a diagram illustrating an example of a screen (Chinese version) used in the voice control operation determination process in a case where the screen includes one slider.

[0026] Fig. 14B This is a diagram showing the structure information of the screen.

[0027] Fig. 14C and Fig.14D is a diagram illustrating an example of voice control operation determination information.

[0028] Fig.14E is an example of a screen used in the voice control operation determination process in the case where the screen includes a slider (with Fig.14A An example diagram of the Japanese version corresponding to the Chinese version shown in FIG.

[0029] Fig.15A and Fig. 15B 1 is a diagram illustrating a processing flow in a case where a plurality of sliders exist in a voice control operation determination unit of an image forming apparatus.

[0030] Fig.16A: is a diagram illustrating an example of a screen (Chinese version) used in the voice control operation determination process in a case where the screen includes a plurality of sliders.

[0031] Fig. 16B This is a diagram showing the structure information of the screen.

[0032] Fig. 16C and Fig.16D is a diagram showing an example of voice control operation determination information.

[0033] Fig.16E is an example of a screen used in the voice control operation determination process when the screen includes a plurality of sliders (with Fig.16A An example diagram of the Japanese version corresponding to the Chinese version shown in FIG.

[0034] FIG. 17A to FIG. 17D This is a sequence diagram illustrating a sequence performed between the system and the user when one slider is provided.

[0035] 18A to 18F This is a sequence diagram illustrating a sequence performed between the system and the user when a plurality of sliders are provided.

[0036] FIG. 19A to FIG. 19C is a diagram illustrating a list of setting items of the image forming apparatus.

[0037] FIG. 20A to FIG. 20C is a diagram illustrating a list of setting items of the image forming apparatus.

[0038] FIG. 21A to FIG. 21C is a diagram illustrating a list of setting items of the image forming apparatus.

[0039] FIG. 22A to FIG. 22F is a diagram illustrating an example of a way of moving a slider.

[0040] FIG. 23A to FIG. 23D is a diagram illustrating an example of a way of moving a slider.

[0041] Fig.24 is a diagram illustrating a slider-shaped operation object.

[0042] FIG. 25A to FIG. 25I is a diagram illustrating an example of a group ID list managed and used by a control program.

[0043] Fig.26 : is a diagram illustrating a processing flow of a voice control operation determination unit of the image forming apparatus.

[0044] Fig.27A : is a diagram showing an example of a Chinese version screen for explaining a voice control operation determination process.

[0045] Fig.27B: is a diagram illustrating another example of the Chinese version screen for explaining the voice control operation determination process.

[0046] Fig.27C This is a diagram illustrating screen configuration information.

[0047] Fig.27D is a diagram illustrating an example of voice control operation determination information.

[0048] Fig.27E : is a diagram showing an example of a Japanese version screen for explaining a voice control operation determination process.

[0049] FIG. 28A to FIG. 28D is a diagram illustrating an example of interaction between a system and a user.

[0050] FIG. 29A to FIG. 29C is a diagram illustrating an example of a way of displaying a focus.

[0051] Fig. 30A 2 is a diagram illustrating a method of moving a focus placed at a default position.

[0052] Fig. 30B 2 is a diagram illustrating a method of moving a focus displayed at a position designated by a user. DETAILED DESCRIPTION

[0053] The present disclosure is described in detail below with reference to the embodiments in conjunction with the accompanying drawings. Note that the components described in the embodiments are merely examples and are not intended to limit the scope of the present disclosure.

[0054] First embodiment

[0055] The first embodiment described below discloses a method of controlling a setting screen by voice for a copy function of an image forming apparatus as an example of an image processing apparatus used in an image processing system. In particular, a method of controlling a setting screen for changing a setting value using an object such as a slider by voice is described.

[0056] System Configuration

[0057] Figure 1 1 is a diagram illustrating an example of a system configuration according to the present embodiment. Figure 1 As shown in , the system includes, for example, a voice control apparatus 100 , an image forming apparatus 101 , a server 102 , a client terminal 103 , and a gateway 105 .

[0058] The voice control apparatus 100, the image forming apparatus 101, and the client terminal 103 can communicate with each other via the gateway 105 and the network 104. Note that the configuration of the system is not limited to including one for each of the voice control apparatus 100, the image forming apparatus 101, and the client terminal 103 connected to each other, but the system may be configured to include two or more for some or each of the voice control apparatus 100, the image forming apparatus 101, and the client terminal 103. Note that the voice control apparatus 100, the image forming apparatus 101, and the client terminal 103 can communicate with the server 102 via the gateway 105 and the Internet.

[0059] The voice control device 100 (voice input device) acquires the voice uttered by the user 106, and transmits the encoded voice data to the server 102 according to the voice control operation start instruction given by the user 106. The voice control device 100 is, for example, a smart speaker. In the configuration of this example, the voice control device 100 and the image forming device 101 are independent separate devices. However, the configuration is not limited to this example. For example, the hardware unit of the voice control device 100 (in Figure 3 described as hardware blocks) and / or software functions (described in Figure 6 Some of the components described as software blocks in the image forming apparatus 101 may be included in the image forming apparatus 101.

[0060] The image forming apparatus 101 (information processing apparatus, image processing apparatus) is a multi-function peripheral device having multiple functions such as a copy function, a scan function, a print function, a fax function, etc. The image forming apparatus 101 may be an apparatus having a single function such as a printer or a scanner. Figure 2 and Figure 5 The operation panel 509 will be described. In the following description, it is assumed by way of example that the image forming apparatus 101 is a color laser beam multifunction peripheral.

[0061] The server 102 (information processing device) performs voice recognition on the voice data of the user 106 acquired by the voice control device 100, and determines words related to the setting operation and job execution of the image forming device 101 based on the result of the voice recognition. In addition, the image forming device 101 generates text based on the voice recognition result or the word determination result, and synthesizes voice data used by the voice control device 100 in reproducing the voice representing the content of the text. Note that a job is a unit of a series of image forming processes (e.g., copying, scanning, printing, etc.) implemented by the image forming device 101 using the print engine 513 and the scanner 515.

[0062] The client terminal 103 is, for example, a personal computer (PC) used by the user 106. The client terminal 103 issues a print job for printing an electronic file by the image forming apparatus 101. The electronic file may be stored in the client terminal 103, or may be stored on a server (not shown) on the Internet, or may be stored in an external storage device 505 of the image forming apparatus 101. The client terminal 103 is also capable of receiving image data obtained by the image forming apparatus 101 via scanning. The operation of the client terminal 103 is not directly related to the subject matter of the present embodiment, and thus further detailed description is omitted.

[0063] The network 104 connects the voice control apparatus 100, the image forming apparatus 101, the client terminal 103, and the gateway 105 to each other. The network 104 transmits and receives various data, such as voice data acquired by the voice control apparatus 100 and transmitted to the server 102, various data transmitted from the server 102, and various data related to a print job, a scan job, and the like.

[0064] The gateway 105 is, for example, a wireless LAN router conforming to the IEEE 802.11 standard series. The gateway 105 may have the ability to operate according to other wireless communication schemes. The gateway 105 may be a wired LAN router conforming to Ethernet standards represented by 10BASE-T, 100BASE-T, 1000BASE-T, etc., instead of a wireless LAN router. The gateway 105 may have the ability to operate according to other wired communication methods. Note that the IEEE 802.11 standard series includes a series of standards such as IEEE 802.11a and IEEE 802.11b belonging to IEEE 802.11.

[0065] Operation panel of image forming apparatus

[0066] Figure 2 is a diagram illustrating an example of the configuration of the operation panel 509 of the image forming apparatus 101 .

[0067] The operation panel 509 includes an LED (Light Emitting Diode) and an LCD (Liquid Crystal Display) display, and functions as a display unit that displays the content of an operation performed by the user 106 and also displays the internal state of the device. The operation panel 509 also functions as a receiving unit that receives an operation performed by the user 106. The operation panel 509 includes not only a plurality of hardware keys but also a touch panel 200 integrated with the LCD display.

[0068] The LCD touch panel 200 is a display that allows touch operation. The LCD touch panel 200 is used to set a mode and display a status.

[0069] The hardware keys 201 to 210 are input units for inputting information. For example, a touch switch is used as the hardware keys 201 to 210.

[0070] The numeric key 201 is a key for inputting a numerical value from 0 to 9. The ID key 202 is a key used when performing an authentication operation (such as a log-in / log-out operation) in the case of using a device protected by user / department authentication management.

[0071] The reset key 203 is a key for resetting the setting mode. The guide key 204 is a key for displaying an explanation screen of each mode. The user mode key 205 is a key for entering a user mode screen. The interrupt key 206 is a key for interrupting copying.

[0072] The start key 207 is a key for starting a copy operation. The stop key 208 is a key for stopping a copy job being executed.

[0073] The power switch 209 is a soft switch. When the power switch 209 is pressed, the backlight of the LCD touch panel 200 is turned off, and the device enters a low power state. The power saving key 210 is a key. When the key is pressed, the device enters a power saving state. When the key is pressed again, the state returns from the power saving state.

[0074] The adjustment key 214 is a key for adjusting the contrast of the LCD touch panel 200 .

[0075] When the counter confirmation key 215 is pressed, a count screen is displayed on the LCD touch panel 200 and indicates the total number of copies that have been made up to that time.

[0076] LED 216 is used to indicate that a job is being executed and an image is being stored in the image memory. LED 217 is an error LED used to indicate that the device is in an error state such as a paper jam or a door open state. LED 218 is a power LED used to indicate that the main switch of the device is in the ON state.

[0077] Hardware structure of voice control device

[0078] Figure 3 is a diagram illustrating an example of the hardware configuration of the controller unit 300 of the voice control apparatus 100 and devices included in the voice control apparatus 100 .

[0079] like Figure 3As shown in FIG. 3 , the controller unit 300 includes a CPU 302, a RAM 303, a ROM 304, an external storage device 305, a network I / F 306, a microphone I / F 307, an audio controller 309, and a display controller 311. These components are connected to a system bus 301 so that they can communicate with each other. The voice control device 100 includes devices such as a microphone 308, a speaker 310, and an LED 312 that are provided on or attached to the controller unit 300.

[0080] The CPU 302 is a central processing unit that controls the operation of the entire controller unit 300. The RAM 303 is a volatile memory. The ROM 304 is a nonvolatile memory and stores a boot program for starting the CPU 302. The external storage device 305 is a storage device (e.g., an SD card) having a larger capacity than the RAM 303. The external storage device 305 stores a control program executed by the controller unit 300 to control the voice control apparatus 100. Note that the external storage device 305 may be replaced with a flash ROM or the like other than an SD card, or may be replaced with another storage device having the same function as the SD card.

[0081] When the power is turned on, etc., the CPU 302 executes a boot program stored in the ROM 304. The boot program is used to read out the control program stored in the external storage device 305 and load it in the RAM 303. After the CPU 302 executes the boot program, the CPU 302 then executes the control program loaded in the RAM 303, thereby performing control. The CPU 302 stores data in the RAM 303 and reads data from and writes data to the RAM 303 when executing the control program. Various setting data and the like required when executing the control program can be stored in the external storage device 305. Various setting data are read and written by the CPU 302. The CPU 302 communicates with other devices on the network 104 via the network I / F 306.

[0082] The network I / F 306 is configured to include a circuit, an antenna, etc. for communicating according to a wireless communication method conforming to the IEEE 802.11 standard series. However, communication is not limited to wireless communication. A wired communication method conforming to the Ethernet standard may be used instead of a wireless communication method.

[0083] The microphone I / F 307 is connected to the microphone 308 , and converts a voice uttered by the user 106 , which is input via the microphone 308 , into encoded voice data (voice information), which is stored in the RAM 303 according to an instruction from the CPU 302 .

[0084] The microphone 308 is, for example, a small MEMS microphone, which may be a type mounted on a smartphone or the like. Note that other types of microphones may be used as long as the voice uttered by the user 106 can be acquired. It is preferred that three or more microphones 308 are provided at predetermined positions so that the direction from which the voice uttered by the user 106 comes can be determined. However, the present embodiment can be implemented even with one microphone 308, and the number of microphones does not necessarily need to be three or more.

[0085] The audio controller 309 is connected to the speaker 310. Under the control of the CPU 302, the audio controller 309 converts the voice data into an analog voice signal, and outputs voice / sound corresponding to the resultant analog voice signal via the speaker 310.

[0086] The speaker 310 reproduces (notifies) a device response sound indicating that the voice control device 100 is responding, and also reproduces the voice synthesized by the server 102. The speaker 310 is a general device for reproducing voice / sound, and its mechanism is not the subject of this embodiment, so further description thereof is omitted.

[0087] The display controller 311 is connected to the LED 312, and controls the display of the LED 312 according to an instruction given by the CPU 302. The display controller 311 mainly controls to turn on the LED 312 to indicate that the voice control apparatus 100 is correctly inputting the voice uttered by the user 106.

[0088] The LED 312 is, for example, a blue LED or the like visible to the user 106. The LED 312 is a general device and is not the subject of the present embodiment, so further description thereof is omitted. Note that a display device capable of displaying characters or pictures may be used instead of the LED 312.

[0089] Server hardware configuration

[0090] Figure 4 is a diagram illustrating an example of a hardware configuration of a controller unit of the server 102 .

[0091] like Figure 4 As shown in FIG. 4 , the controller unit includes a CPU 402 , a RAM 403 , a ROM 404 , an external storage device 405 , and a network I / F 406 connected to a system bus 401 .

[0092] The CPU 402 is a central processing unit that controls the operation of the entire controller unit. The RAM 403 is a volatile memory. The ROM 404 is a non-volatile memory, and a boot program for starting the CPU 402 is stored in the ROM 404. The external storage device 405 is a storage device (e.g., a hard disk drive (HDD)) having a larger capacity than the RAM 403. The external storage device 405 stores a control program executed by the controller unit to control the server 102. Note that the external storage device 405 can be replaced with another storage device having the same function as the hard disk drive. For example, a solid state drive (SSD) or the like can be used.

[0093] When the power is turned on, etc., the CPU 402 executes a boot program stored in the ROM 404. The boot program is used to read out the control program stored in the external storage device 405 and load it in the RAM 403. After the CPU 402 executes the boot program, the CPU 402 then executes the control program loaded in the RAM 403, thereby performing control. When the CPU 402 executes the control program, it stores data in the RAM 403 and reads data from the RAM 403 and writes data to the RAM 403. The external storage device 405 can further store various settings required when executing the control program. Various setting data are read and written by the CPU 402. The CPU 402 communicates with other devices on the network 104 via the network I / F 406.

[0094] Hardware structure of image forming device

[0095] Figure 5 is a diagram illustrating an example of the hardware configuration of the controller unit 500 of the image forming apparatus 101 and devices included in the image forming apparatus 101 .

[0096] like Figure 5 As shown in FIG. 5 , the controller unit 500 includes a CPU 502, a RAM 503, a ROM 504, an external storage device 505, a network I / F 506, a display controller 507, an operation I / F 508, a print controller 512, and a scan controller 514. These components are connected to a system bus 501 so that they can communicate with each other.

[0097] The CPU 502 is a central processing unit that controls the operation of the entire controller unit 500. The RAM 503 is a volatile memory. The ROM 504 is a nonvolatile memory, and a boot program for starting the CPU 502 is stored in the ROM 504. The external storage device 505 is a storage device (e.g., a hard disk drive (HDD)) having a larger capacity than the RAM 503. The external storage device 505 stores a control program executed by the CPU 502 to control the image forming apparatus 101. Note that the external storage device 505 can be replaced with another storage device having the same function as the hard disk drive. For example, a solid state drive (SSD) or the like can be used.

[0098] When the power is turned on, etc., the CPU 502 executes a boot program stored in the ROM 504. The boot program is used to read out the control program stored in the external storage device 505 and load it in the RAM 503. After the CPU 502 executes the boot program, the CPU 502 then executes the control program loaded in the RAM 503, thereby performing control. When the CPU 502 executes the control program, it stores data in the RAM 503 and reads data from and writes data to the RAM 503. The external storage device 505 can further store various setting data required when executing the control program and image data read by the scanner 515. These data are read and written by the CPU 502. The CPU 502 communicates with other devices on the network 104 via the network I / F 506.

[0099] The display controller 507 controls the display of the screen of the LCD touch panel 200 of the connected operation panel 509 under the control of the CPU 502 .

[0100] The operation I / F 508 inputs and outputs operation signals. The operation I / F 508 is connected to the operation panel 509. When the LCD touch panel 200 is pressed, the CPU 502 acquires the coordinates of the pressed position on the LCD touch panel 200 via the operation I / F 508. The operation I / F 508 also detects the pressing of the hardware keys 201 to 210 on the operation panel 509.

[0101] The print controller 512 sends control commands and image data to the connected print engine 513 in response to instructions issued by the CPU 502 .

[0102] The print engine 513 prints the received image data on a sheet according to a control command received from the print controller 512. The details of the print engine 513 are not the subject of the present embodiment, and thus further description thereof is omitted.

[0103] The scanning controller 514 sends a control command to the connected scanner 515 according to an instruction issued by the CPU 502 , and writes image data received from the scanner 515 in the RAM 503 .

[0104] The scanner 515 reads a document placed on a platen glass (not shown) of the image forming apparatus 101 using an optical unit according to a control command received from the scanning controller 514. A detailed description of the scanner 515 is not the subject of the present embodiment, and thus is omitted.

[0105] Functional structure of voice control program of voice control device

[0106] Figure 6 302 is a block diagram showing a functional configuration of a voice control program 601 of the voice control apparatus 100 executed by the CPU 302.

[0107] The voice control program 601 of the voice control apparatus 100 is stored in the external storage device 305 as described above. When the voice control program 601 is started, the voice control program 601 is loaded into the RAM 303 by the CPU 302 and executed.

[0108] The data sending and receiving unit 602 sends and receives data to and from other devices on the network 104 through TCP / IP via the network I / F 306. The data sending and receiving unit 602 sends voice data of a voice uttered by the user 106, which is acquired by the voice acquisition unit 604 described later, to the server 102. In addition, the data sending and receiving unit 602 receives synthesized voice data (synthesized voice) generated by the server 102 as a response to the user 106.

[0109] The data management unit 603 stores various data such as work data generated in the execution of the voice control program 601 in a predetermined area of ​​the external storage device 305, and manages the data. Examples of such data stored and managed include volume setting data related to the voice reproduced by the voice reproduction unit 605 described later, authentication information required for communication with the gateway 105, and device information required for communication with the image forming apparatus 101 and the server 102.

[0110] The voice acquisition unit 604 converts the analog voice of the user 106 near the voice control device 100 acquired by the microphone 308 into voice data and temporarily stores it. The voice of the user 106 is converted into a predetermined format such as MP3, and is temporarily stored in the RAM 303 as encoded voice data to be transmitted to the server 102. The start timing and end timing of the processing of the voice acquisition unit 604 are controlled by the voice control unit 609. The voice data may be encoded into the encoded voice data in a general stream format, and the encoded voice data may be sequentially transmitted by the data transmission and reception unit 602.

[0111] The voice reproducing unit 605 reproduces the synthesized voice data received by the data transmitting and receiving unit 602 through the speaker 310 via the audio controller 309. The timing at which the voice reproducing unit 605 reproduces the voice is controlled by the voice control unit 609 as described later.

[0112] The display unit 606 controls the LED 312 to turn on / off via the display controller 311. For example, when the voice control operation start detection unit 607 (described later) detects that the voice control operation is in progress, the LED 312 is turned on. The timing at which the display unit 606 turns on the LED 312 is managed by the voice control unit 609 described later.

[0113] When the voice control operation start detection unit 607 detects a wake-up word uttered by the user 106 or detects the pressing of an operation start key (not shown) of the voice control device 100, the voice control operation start detection unit 607 sends an operation start notification to the voice control unit 609. Note that the wake-up word is a predetermined voice word. The voice control operation start detection unit 607 always performs an operation of detecting the wake-up word from the simulated voice of the user 106 near the voice control device 100 acquired by the microphone 308. The user 106 is allowed to operate the image forming device 101 by uttering the wake-up word and then speaking to indicate what operation is to be performed. When the wake-up word is detected by the voice control operation start detection unit 607, voice processing as will be described later is performed.

[0114] The utterance end determination unit 608 detects the time when the processing of the voice acquisition unit 604 ends. For example, when the user 106 does not utter a voice within a certain period of time (for example, 3 seconds), it is determined that the utterance of the user 106 has ended, and a notification of the end of utterance is sent to the voice control unit 609. Note that the determination of the end of utterance can be made based on a predetermined phrase uttered by the user 106, rather than based on a period of time when no utterance occurs (hereinafter referred to as a blank time). For example, the predetermined phrase can be "yes", "no", "ok", "cancel", "end", "start", "start", etc. In the case of uttering such a predetermined phrase, it can be determined that the utterance has ended without waiting for a predetermined period of time. The determination of the end of utterance can be made by the server 102 instead of by the voice control device 100. The end of utterance can be determined based on the meaning or context of the utterance of the user 106.

[0115] The voice control unit 609 plays a central role in the processing. The voice control unit 609 controls the modules in the voice control program 601 to operate in cooperation with each other. More specifically, the voice control unit 609 controls the start and end of the processing performed by the voice acquisition unit 604, the voice reproduction unit 605 and the display unit 606. In addition, when the voice acquisition unit 604 acquires voice data, the voice control unit 609 controls so that the voice data is sent to the server 102 by the data transmission and reception unit 602. In addition, when the data transmission and reception unit 602 receives the synthesized voice data from the server 102, the voice control unit 609 controls so that the synthesized voice data is reproduced by the voice reproduction unit 605.

[0116] Next, a description is given below about the timing of the start and end of processing performed by the voice acquisition unit 604, the voice reproduction unit 605, and the display unit 606.

[0117] When the voice control unit 609 receives the operation start notification from the voice control operation start detection unit 607, the voice control unit 609 starts the processing of the voice acquisition unit 604. In the case where the voice control unit 609 receives the utterance end notification from the utterance end determination unit 608, the voice control unit 609 ends the processing of the voice acquisition unit 604. For example, in the case where the user 106 utters the wake-up word and then says "I want to make a copy", the following processing is performed. In this case, when the voice control operation start detection unit 607 detects the voice of the wake-up word, the voice control operation start detection unit 607 sends the operation start notification to the voice control unit 609. When the voice control unit 609 receives the operation start notification, the voice control unit 609 controls to start the processing of the voice acquisition unit 604. The voice acquisition unit 604 converts the subsequent simulated voice saying "I want to make a copy" into voice data, and temporarily stores the resulting voice data. In the case where the utterance end determination unit 608 determines that a non-utterance period of a predetermined length has occurred after the utterance of "I want to make a copy", the utterance end determination unit 608 sends an utterance end notification to the voice control unit 609. When the voice control unit 609 receives the utterance end notification, the voice control unit 609 ends the processing of the voice acquisition unit 604. Note that the state from the start of the processing of the voice acquisition unit 604 to its end is referred to as the utterance processing state. The display unit 606 turns on the LED 312 to indicate that the processing is currently in the utterance processing state.

[0118] When it is determined that the utterance has ended, the voice control unit 609 controls the data transmission and reception unit 602 to transmit voice data to the server 102, and waits for a response to be received from the server 102. The response from the server 102 is given in the form of, for example, a response message, which includes a header indicating that the response is a response and synthesized voice data. When the voice control unit 609 receives the response message via the data transmission and reception unit 602, the voice control unit 609 controls the voice reproduction unit 605 to reproduce the synthesized voice data. The synthesized voice data, for example, indicates "a copy screen will be displayed." The state from the end of the utterance to the end of the reproduction of the synthesized voice data is called the response processing state. The display unit 606 flashes the LED 312 to indicate that the processing is currently in the response processing state.

[0119] In the period of continuing the interactive session with the server 102 after the response processing, the user 106 is allowed to speak what the user wants to do without issuing a wake-up word. The server 102 determines whether the interactive session is completed. When the server 102 determines that the interactive session is over, the server 102 sends an interactive session end notification to the voice control device 100. Note that the state from the end of the interactive session to the start of the next interactive session is called a waiting state. It is assumed that the processing state is always in the waiting state before the voice control device 100 receives the operation start notification from the voice control operation start detection unit 607. The display unit 606 controls so that the LED 312 is in the off state during the waiting state.

[0120] Functional structure of the voice data conversion control program of the server

[0121] Fig. 7A 4 is a block diagram illustrating the configuration of a voice data conversion control program 701 performed by the CPU 402 in the server 102. Figure 7B An example of a group ID list used in determining a group ID by the group ID determination unit 707 is illustrated. In the group ID list, words having the same meaning or intention with respect to the user operation of the image forming apparatus 101 are grouped and the same ID is assigned. Note that the words are obtained by performing voice recognition on an utterance given by the user 106 to the voice control apparatus 100.

[0122] The speech data conversion control program 701 for the server 102 is stored in the external storage device 405 as described above, and is loaded into the RAM 403 and executed by the CPU 402 when the program is started.

[0123] The data transmission and reception unit 702 transmits and receives data to and from other devices on the network 104 using TCP / IP via the network I / F 406. The data transmission and reception unit 702 receives the voice data of the user 106 from the voice control device 100. In addition, the data transmission and reception unit 702 also transmits the result of group ID determination performed by the group ID determination unit 707 as will be described later.

[0124] The data management unit 703 stores various data (such as working data generated in the execution of the voice data conversion control program 701 and parameters required for the voice data converter 704 to perform voice recognition processing) in a determined storage area of ​​the external storage device 405, and the data management unit 703 manages these various data. Some examples of such data are described below. The acoustic model and language model used by the voice recognition unit 705 in converting the voice data received by the data sending and receiving unit 702 into text data (text information) are stored in a specific storage area and managed. The dictionary used by the morphological analysis unit 706 in performing morphological analysis of the text is stored in a predetermined area of ​​the external storage device 405 and managed. The group ID list used by the group ID determination unit 707 to determine the group ID is stored in a predetermined area of ​​the external storage device 405 and managed. The voice database used by the voice synthesis unit 708 in performing voice synthesis is stored in a predetermined area of ​​the external storage device 405 and managed. Furthermore, in the data management unit 703 , device information necessary for communicating with the voice control apparatus 100 and the image forming apparatus 101 is stored and managed.

[0125] The speech data converter 704 includes a speech recognition unit 705, a morphological analysis unit 706, a group ID determination unit 707, and a speech synthesis unit 708. The speech data converter 704 is described in further detail below.

[0126] The speech recognition unit 705 performs speech recognition processing for converting the speech data of the user 106 received by the data sending and receiving unit 702 into text. In the speech recognition processing, the speech data of the user 106 is converted into phonemes using an acoustic model, and the phonemes are further converted into actual text data by pattern matching using a language model. The acoustic model can be a model of a machine learning method using a neural network such as DNN-HMM, or a model of another method such as GMM-HMM. In machine learning using a neural network, learning of the learning model is performed based on training data including, for example, pairs of speech and text. The language model can be a model of a machine learning method using a neural network (such as RNN), or a model of another method such as an N-gram method.

[0127] In the present embodiment, text data includes the text that comprises one or more kana and the text that is obtained by performing " kana-kanji conversion " (including conversion into numbers, letters, symbols, etc.). However, the speech recognition process that is used to convert voice data into text data is not limited to the above-mentioned method, and other methods can be used. The details of speech recognition process are not the subject of the present embodiment, and therefore further description is omitted.

[0128] The morphological analysis unit 706 performs morphological analysis on the text data converted by the speech recognition unit 705. In the morphological analysis, a morpheme sequence is derived from a dictionary having information related to the grammar, part of speech, etc. of the language, and the part of speech of each morpheme is determined. The morphological analysis unit 706 can be implemented using well-known morphological analysis software such as JUMAN, ChaSen, MeCab, etc. For example, when text data representing "I want to make a copy" is given as a result of the conversion of the speech recognition unit 705, the morphological analysis unit 706 analyzes the text data into a morpheme sequence of "I", "want", "want", "make", and "copy". In the case of giving text data representing "from A3 to A4", the morphological analysis unit 706 analyzes the text data into a morpheme sequence of "from", "A3", "to", and "A4".

[0129] The group ID determination unit 707 determines the group ID by matching the result of the morphological analysis of the morphological analysis unit 706 with the group ID list in FIG. 7B , and further generates a group ID determination result. For example, from the morpheme sequence of "kopi (copy)", "wo", "shi", "tai", "FNC00001" is detected as the group ID of "kopi (copy)", and {ID: FNC00001} is generated as the result of the group ID determination. From the morpheme sequence of "A3", "kara", "A4", "he", two group IDs "PAP00100" and "PAP00101" are detected as the group IDs of "A3" and "A4", and {ID: PAP00100, ID: PAP00101} is generated as the result of the group ID determination.

[0130] In the case where multiple IDs are generated as a result of group ID determination, the group ID determination result is generated in the order of speech recognition and morphological analysis. For example, in the case where "A4", "kara", "A3" and "he" are obtained in this order as a result of speech recognition and morphological analysis, {ID: PAP00101, ID: PAP00100} is generated as the result of group ID determination. Note that multiple adjacent morphemes can be combined and matched with the group ID list, and determination can be made based on the matching result. In this case, if a morpheme matching the group ID in the group ID list is found, and in addition, multiple morphemes including the first morpheme are found to match the group ID in the group ID list, the latter result is used in generating the group ID determination result. For example, in the case where the morpheme sequence "A", "4" is given, the group ID determination result is not generated as {ID: CHR00000, ID: NUM00004}, but is generated as {ID: PAP00101}. Note that it is possible to match so that the text comprising one or more kana obtained as a result of speech recognition and morphological analysis is combined with the text obtained as a result of carrying out "kana-kanji conversion", and the combination of gained is matched with the group ID list. For example, first, group ID determining unit 707 matches between the text obtained via "kana-kanji conversion" and the "kana-kanji" text in the group ID list. When the group ID that is not found to be matched is matched between the kana text obtained as a result of speech recognition and morphological analysis and the "kana" text in the group ID list, to detect the group ID for the kana text that is matched. Multiple different group IDs in the group ID list are assigned to determine ID at identical "kana", and therefore multiple group IDs match, multiple group IDs can be generated to determine the result as a candidate. This enables the generation of a group ID that takes into account the error in "kana-kanji conversion" and the spelling difference of Chinese characters.

[0131] The speech synthesis unit 708 performs speech synthesis processing based on the notification received from the image forming apparatus 101. In the speech synthesis processing, the text corresponding to the notification is converted into speech data in a predetermined format such as MP3. 9A to 9CAn example of a combination of received notification data and text to be speech synthesized is described with reference to the sequence diagram in . In speech synthesis processing, for example, speech data is generated based on a speech database stored in the data management unit 703. The speech database is, for example, a database including a collection of speech uttered for fixed words or phrases. In the present embodiment, speech synthesis processing is performed using a speech database. However, the technology used in speech synthesis processing is not limited to this example, and another technology may be used. The details of speech synthesis processing are not the subject of the present embodiment, and therefore further description thereof is omitted. Functional structure of device control program of image forming apparatus

[0132] Figure 8 801 is a block diagram illustrating a functional configuration of a device control program 801 of the image forming apparatus 101 executed by the CPU 502 .

[0133] The device control program 801 of the image forming apparatus 101 is stored in the external storage device 505 as described above, and is loaded into the RAM 503 and executed by the CPU 502 at the time of activation.

[0134] The data transmission and reception unit 802 transmits and receives data to and from other devices on the network 104 through TCP / IP via the network I / F 506. More specifically, for example, the data transmission and reception unit 802 receives the group ID determination result generated by the group ID determination unit 707. In addition, the data transmission and reception unit 802 transmits a screen update notification indicating that the screen display content displayed on the LCD touch panel 200 on the operation panel 509 has been updated and a job execution status notification indicating the status of the job from the image forming apparatus 101 to the server 102. This will be referred to later. 9A to 9C The content of the notification is described using the sequence diagram shown in .

[0135] The data management unit 803 stores and manages various data, such as work data generated in the execution of the device control program 801 and setting parameters required to control various devices, in predetermined areas of the RAM 503 and the external storage device 505. For example, job data including a combination of setting items and corresponding setting values ​​of a job executed by the device control unit 808 described later and machine setting information for setting paper attribute information and the like are stored and managed. In addition, authentication information required for communication with the gateway 105, device information required for communication with the server 102, and the like are stored and managed. In addition, image data processed by the image forming apparatus 101 are stored and managed. In addition, screen control information used by the display unit 806 in performing screen display control and voice control operation determination information used by the voice control operation determination unit 807 in determining an operation are stored and managed. The screen control information and the voice control operation determination information are managed separately for each screen displayed by the display unit 806.

[0136] The scanning unit 804 performs scanning using the scanner 515 via the scanning controller 514 based on the scan job parameter setting of the device control unit 808 , and stores the resulting scanned image data in the data management unit 803 .

[0137] The printing unit 805 executes printing using the print engine 513 via the print controller 512 based on the print job parameter setting of the device control unit 808 .

[0138] The display unit 806 controls the operation panel 509 via the display controller 507, and displays user-operable UI components (buttons, drop-down lists, check boxes, etc.) on the LCD touch panel 200 based on the screen display control information. In addition, the display unit 806 obtains the touch coordinates on the LCD touch panel 200 (hereinafter referred to as the screen, etc.) via the operation I / F 508, and determines the UI component to be operated and the processing to be performed when the operation of the UI component is received. In addition, detection is performed on the pressing of the hardware keys 201 to 210 on the operation panel 509. According to the result thereof, the content displayed on the screen is updated, and the job parameters set by the user operation and the instruction to start the job are sent to the device control unit. Similarly, according to the result of the voice control operation determination of the voice control operation determination unit 807 described later, the content displayed on the screen is updated, and the job parameters set by the user operation and the instruction to start the job are sent to the device control unit.

[0139] The voice control operation determination unit 807 determines one of the user-operable UI components presented on the screen displayed on the operation panel 509 or one of the hardware keys 201 to 210 provided on the operation panel 509 as an operation target based on the group ID determination result received by the data transmission and reception unit 802. FIG. 11A to FIG. 11H Details of the voice control operation determination unit 807 are described.

[0140] The device control unit 808 issues control instructions to the print engine 513 and the scanner 515 via the print controller 512 and the scan controller 514. For example, in the case where the display unit 806 detects that the start key 207 has been pressed when the copy function screen is displayed, the device control unit 808 receives the copy job parameters and the job start instruction from the display unit 806. Based on the job parameters, control is performed so that the print engine 513 prints the image data scanned by the scanner 515 on a sheet. Note that the scanning and printing control mechanism is not the subject of the present embodiment, and therefore further description thereof is omitted.

[0141] System Control Sequence

[0142] 9A to 9C It is an example of Figure 1 between the devices of the system shown in Figures 6 to 8 More specifically, 9A to 9C A sequence is illustrated in which the voice control apparatus 100 receives a voice control operation through a voice uttered by the user 106 , and the image forming apparatus 101 executes various processes according to the voice control operation and returns a voice response indicating the execution result to the user 106 .

[0143] exist 9A to 9C In the example of the sequence shown in , it is assumed that the voice control device 100, the image forming device 101, and the server 102 are in a state where they can communicate with each other. It is also assumed that the image forming device 101 is displaying a main menu screen that enables functions such as a copy function, a scan function, and a print function to be called after the power is turned on.

[0144] First, in step 905 (hereinafter referred to as S905), the user 106 instructs the voice control device 100 to start the voice control operation. The user 106 issues the instruction to start the voice control operation by uttering a wake-up word or pressing the operation start key of the voice control device 100. The voice control operation start detection unit 607 detects the voice control operation start instruction.

[0145] When the voice control operation start instruction is detected, then, in S906, the display unit 606 of the voice control program 601 turns on the LED to indicate that the utterance process is in progress. At the same time, the process of the voice acquisition unit 604 is started.

[0146] In S907, the user 106 issues a function call instruction to the voice control device 100. The function call instruction is issued by the user 106 by uttering, for example, "I want to make a copy" or "open the copy screen" following the wake-up word in S905. Based on the voice acquired by the voice acquisition unit 604, voice data is generated. When a blank time of a predetermined time or longer occurs, the utterance end determination unit 608 determines that the utterance has ended.

[0147] In S908, in response to the utterance end determination, the display unit 606 of the voice control program 601 flashes the LED to indicate that the response process is in progress. At the same time, the process of the voice acquisition unit 604 ends.

[0148] When the voice data of the function calling instruction is generated in S907 , then in S909 , the data transmitting and receiving unit 602 transmits the generated voice data to the server 102 .

[0149] In S910, the speech recognition unit 705 performs speech recognition processing on the speech data received by the data transmission and reception unit 702 of the speech data conversion control program 701. As a result of the speech recognition processing, for example, the speech "I want to make a copy" uttered by the user 106 is converted into text.

[0150] Then, in S911, the morphological analysis unit 706 of the speech data conversion control program 701 performs morphological analysis processing on the text generated in S910. For example, the text "kopishitai (I want to make a copy)" is divided into the morphological sequences "kopi (copy)", "shi (make)" and "tai (want)" as the result of the morphological analysis processing.

[0151] In S912, the group ID determination unit 707 of the speech data conversion control program 701 performs group ID determination processing on the morphological sequence text. In the group ID determination processing, the morpheme sequence "kopi (copy)", "shi (carry out)", "tai (think)" and Figure 7B A matching process is performed between the group ID lists shown in , and {ID: FNC00001} is generated as a group ID determination result.

[0152] In S913 , the data transmission and reception unit 702 of the speech data conversion control program 701 transmits the group ID determination result obtained in S911 to the image forming apparatus 101 .

[0153] In S914, the voice control operation determination unit 807 performs a voice control operation determination process on the group ID determination result received by the data transmission and reception unit 802 of the device control program 801. For example, it is determined from the group ID determination result {ID: FNC00001} that the "Copy" button in the main menu screen has been selected as a result of the voice control operation determination process.

[0154] Thereafter, in S915, the content displayed on the screen by the display unit 806 is updated according to the determination result in S914. For example, in the case where it is determined from the voice "kopishitai (I want to make a copy)" uttered by the user that the voice control operation specifies the operation of the "copy" button displayed on the screen, the copy function screen is displayed in the same manner as in the case of touching the "copy" button on the LCD touch panel 200.

[0155] In S916, the data transmission and reception unit 802 transmits a screen update notification to the server 102 to notify that the content displayed on the screen has been updated. For example, when the display unit 806 changes the display content from the main menu screen to the copy function screen, the display unit 806 transmits text data "Open the copy function screen" as the screen update notification.

[0156] In S917, in response to the screen update notification received by the data transmission and reception unit 702 of the voice data conversion control program 701, the voice synthesis unit 708 performs voice synthesis processing to generate synthesized voice of predetermined text data corresponding to the content of the screen update notification. For example, in the case where the screen update notification indicates "open the copy function screen", the voice synthesis unit 708 synthesizes voice corresponding to the text data "open the copy screen". The voice data (synthesized voice data) generated by the voice synthesis unit 708 in the voice synthesis processing is sent to the voice control device 100 by the data transmission and reception unit 702.

[0157] In S918, the data transmitting and receiving unit 602 receives the synthesized speech data generated and transmitted in S917.

[0158] In S919, the speech reproducing unit 605 reproduces the synthesized speech data received in S918. For example, the synthesized speech data "Open the copy screen" generated in S917 is reproduced through the speaker 310.

[0159] In S920, after the synthesized speech data is reproduced, the display unit 606 of the speech control program 601 turns on the LED again to indicate that the utterance process is in progress. At the same time, the process of the speech acquisition unit 604 is started again.

[0160] In S921, the user 106 issues a setting screen call instruction to the voice control device 100. When the user 106 issues, for example, "select paper", the setting screen call instruction is issued. In response, voice data corresponding to the uttered voice is acquired by the voice acquisition unit 604. When a blank time of a predetermined time or more occurs after the utterance of the user 106, the utterance end determination unit 608 determines that the utterance has ended.

[0161] S922 is similar to the above-mentioned S908.

[0162] In S923, the voice control operation process is performed in a similar manner to the process from S909 to S918 described above. However, in S923, the screen is updated so that the display unit 806 displays the setting screen in response to the setting screen call instruction in S921. For example, when the copy function screen is displayed, if the group ID determination result is {ID: PAP00000, ID: OPR00040}, the paper selection screen is displayed.

[0163] In S924, the speech reproduction unit 605 reproduces the synthesized speech data generated in the speech synthesis processing in S923. For example, if the paper selection screen is displayed as the setting screen in S923, a message "Paper selection screen is displayed. Perform paper setting" is reproduced via the speaker 310.

[0164] The S925 is similar to the S920 described above.

[0165] In S926, the user 106 issues a setting change instruction to the voice control apparatus 100. The setting change instruction is issued by the user 106 by uttering, for example, "A4." When a blank time of a predetermined time or more occurs after the utterance of the user 106, the utterance end determination unit 608 determines that the utterance has ended.

[0166] S927 is similar to the above-mentioned S908.

[0167] In S928, the voice control operation process is performed in a similar manner to the process from S909 to S918 described above. However, in S928, the setting value displayed on the setting screen is changed according to the setting change instruction in S926. For example, when the paper selection screen is displayed, if the group ID determination result is {ID: PAP00101}, the paper selection is changed to A4 on the paper selection screen, and the result is displayed.

[0168] In S929, the speech reproduction unit 605 reproduces the synthesized speech data generated in the speech synthesis processing in S928. For example, in the case where the paper setting value is changed and displayed in S928, the synthesized speech data "the paper has been set to A4" is reproduced via the speaker 310.

[0169] The S935 is similar to the S920 described above.

[0170] In S936, the user 106 issues a job execution instruction to the voice control apparatus 100. The job execution instruction is issued by the user 106 by uttering, for example, "Copy start." When a blank time of a predetermined time or more occurs after the utterance of the user 106, the utterance end determination unit 608 determines that the utterance has ended.

[0171] S937 to S942 are similar to the above-mentioned S909 to S913.

[0172] In S943, the voice control operation determination unit 807 performs voice control operation determination processing on the group ID determination result received by the data transmission and reception unit 802 of the device control program 801. In a case where the group ID determination result is {ID: FNC00001, ID: OPP00011}, it is determined that the “copy start” button or the hard start key 207 displayed on the screen has been operated.

[0173] In S944, the job execution screen is displayed according to the determination result obtained in S943. For example, in the case where it is determined from the voice "copy start" uttered by the user that the operation instructed to be performed is to operate the start key 207, the copy job start screen is displayed.

[0174] In S945 , the job is executed according to the job parameters set on the screen of the image forming apparatus 101 .

[0175] In S946, the data sending and receiving unit 802 sends information indicating that job execution has started (job execution start notification) as the content of job execution status notification to the server 102. For example, in the case where a copy job is started, text data "copy job started" is sent as the content of the job execution status.

[0176] In S947, the data sending and receiving unit 702 of the voice data conversion control program 701 receives the job execution status notification, and the voice synthesis unit 708 performs voice synthesis processing on the predetermined text data corresponding to the content of the job execution status notification (job execution start notification). For example, in the case where the content of the job execution status notification is "copy job start", the voice synthesis unit 708 synthesizes the voice "copy start" based on the text data.

[0177] The S948 is similar to the S918 described above.

[0178] In S949, the speech reproduction unit 605 reproduces the synthesized speech data received in S948. For example, the synthesized speech data "Copy Start" generated in S947 is reproduced via the speaker 310.

[0179] In S950, the data sending and receiving unit 802 sends information indicating the end of job execution (job execution end notification) as the content of the job execution status notification to the server 102. For example, when the copy job is completed, the job execution status notification including the text data "copy job completed" as its content is sent.

[0180] In S951, the data transmission and reception unit 702 of the voice data conversion control program 701 receives the job execution status notification, and the voice synthesis unit 708 performs voice synthesis processing on the predetermined text data corresponding to the content of the job execution status notification (job execution end notification). For example, in the case where the content of the job execution status notification is "copy job completed", the voice synthesis unit 708 synthesizes the voice "copy completed" based on the text data.

[0181] In S952, in response to the end of the job execution processing in S945, the display unit 806 displays a job execution end screen. For example, when the execution of the copy job is completed, the copy job end screen is displayed.

[0182] In S953, the data transmitting and receiving unit 602 receives the synthesized voice data generated in S951 from the server 102. In addition, an interactive session end notification for notifying the voice control apparatus 100 that the interactive session with the user 106 has ended is received from the server 102.

[0183] In S954, the speech reproducing unit 605 reproduces the synthesized speech data received in S953. For example, the synthesized speech data "Copy Completed" generated in S951 is reproduced via the speaker 310.

[0184] In S955, in response to the reception of the interactive session end notification in S953, the display unit 606 of the voice control program 601 turns off the LED to indicate that the voice control apparatus 100 enters a waiting state.

[0185] In S956, in response to the reception of the interactive session end notification in S953, the voice control apparatus 100 switches to a waiting state.

[0186] Note that even when the LED is in a state on the sequence diagram where the LED is flashing to indicate that response processing is in progress, input of the wake word is allowed at any time. The user 106 can forcefully end the interactive session by uttering "cancel" or "stop" following the utterance of the wake word.

[0187] Processing flow of the device control program 801 of the image forming apparatus 101

[0188] Fig.10 1 is a flowchart illustrating an outline of the processing flow of the device control program 801 of the image forming apparatus 101. More specifically, Fig.10 Shows 9A to 9C An example of the processing flow of the device control program 801 in S914, S915, S923, S928, S933, and S943 to S952 in the sequence diagram shown in .

[0189] FIG. 11A to FIG. 11H and FIG. 12A to FIG. 12D Will be used to show Fig.10 A specific example of the processing flow shown in . FIG. 11A to FIG. 11H is added to Figure 7B Another example of a group ID list is a group ID list shown in . Fig. 12A The screen displayed by the display unit 806 is illustrated, wherein the screen is from Fig.12D The Chinese version of the translation of the Japanese version shown in . Fig. 12B Illustrated with Fig. 12A An example of screen control information corresponding to the screen shown in . The screen control information indicates the correspondence between the user-operable UI components displayed on the screen and the actions taken in response to operating the user-operable UI components (the contents of the screen control and the internal processing to be performed). Fig. 12C Shown with Fig. 12A An example of voice control operation determination information corresponding to the screen shown in . Fig.12D The screen displayed by the display unit 806 is illustrated. Fig. 12A The Chinese version shown in corresponds to the Japanese version. Note that the voice control operation determination information indicates a predefined correspondence between a user-operable UI component and one or more group ID determination results. Fig. 12C Among the pieces of information described in the table shown in , “examples of phrases uttered by the user in operation” and “operations performed on the operation panel” are used in the description given below, but they do not necessarily need to be included in the voice control operation determination information managed by the data management unit 803. Fig. 12A In the example shown in , the screen includes a slider-shaped operation object. Fig.242 is a diagram illustrating the details of a slider-shaped operation object. In the present embodiment, operation objects such as sliders, slide bars, scroll bars, track bars, etc. are referred to as slider-shaped operation objects. The setting value and setting state are represented by the position of a knob (roller, bubble box, thumbnail, scroll box, knob, lifter, quint, pack, wiper, handle, etc.). The position of the knob can be moved by pressing a move button (arrow button) or dragging the knob. Both or only one of the above-mentioned moving methods may be available. A ruler scale may be provided or not provided on the main body of the slider-shaped operation object. Fig. 12C The example shown in FIG. 12D is applicable to a voice control operation performed by the user viewing the Japanese version screen shown in FIG. 12D .

[0190] In S1001 , the data transmitting and receiving unit 802 receives the group ID determination result generated by the group ID determination unit 707 .

[0191] In step S1002, screen control information corresponding to the screen displayed by the display unit 806 and voice control operation determination information used by the voice control operation determination unit 807 to determine the operation target are acquired. Fig. 12A In the case of the screen shown in Fig. 12A The screen shown in Fig. 12B The screen control information shown in Fig. 12C The voice control operation confirmation information shown in .

[0192] In S1003, the voice control operation determination unit 807 uses the group ID determination result received in S1001 and the voice control operation determination information obtained in S1002 to perform a voice control operation determination process. In the voice control operation determination process, one of the user-operable UI components displayed on the screen of the operation panel 509 or one of the hardware keys 201 to 210 provided on the operation panel 509 is determined as an operation target. For example, in the case where the user 106 issues, for example, "auto", the group ID determination unit 707 generates {ID: DRG00008} as a group ID determination result, and the group ID determination result is sent. The data sending and receiving unit 802 receives the group ID determination result in S1001, and the voice control operation determination unit 807 detects that the "auto (color / black and white)" button matches the received group ID determination result, so the voice control operation determination unit 807 determines that the "auto (color / black and white)" button is the operation target.

[0193] Will refer to later FIG. 14A to FIG. 14E and other drawings to describe the detailed flow of the voice control operation determination process and an example of the determination process.

[0194] In S1004, the display unit 806 updates the content displayed on the screen based on the result of the voice control operation determination processing in S1003 and the screen control information acquired in S1002. For example, if the "Auto (Color / Black and White)" button is determined as the operation target in S1003, then based on Fig. 12B The screen control information shown in is used to update the screen so that the display unit 806 highlights the button determined as the operation target. Fig. 12C In the example shown in , the voice control operation determination information indicates that when the “Auto (Color / Black and White)” button displayed on the LCD touch panel 200 of the operation panel 509 is touched, the button is also highlighted.

[0195] If the result of the voice control operation determination processing in S1003 is a voice instruction indicating transition to the slider screen, the display is transitioned to the corresponding slider screen. If the result of the voice control operation determination processing in S1003 is an instruction to move the slider as described later in S1309 to S1315, the following processing is performed. That is, the displayed screen is modified so that the pointing scale scale (pointing position) of the slider is based on Fig. 14B The screen control information shown in Fig. 14C In this process, the pointing position of the slider is changed based on the pointing position of the slider when the slider is read in S1306 as described later. After the setting by the slider is completed, the setting parameters of the data management unit 803 of the device control program 801 are updated.

[0196] In S1005, a determination is made as to whether to execute the job based on the result of the voice control operation determination processing in S1003 and the screen control information acquired in S1002. When the job is to be executed, the processing proceeds to S1007, otherwise the processing proceeds to S1006. For example, in the case where the "Auto (color / black and white)" button is determined as the operation target in S1003, since there is no internal processing to be performed in FIG. 12B, it is determined that the job is not to be executed. In the case where it is determined in S1003 that the start key 207 is the operation target, since in this case Fig. 12B The internal processing defined in is "start execution of a copy job", so it is determined that the job execution is to be performed.

[0197] In S1006 , as shown in S916 described above, the data sending and receiving unit 802 sends a screen update notification to the server 102 .

[0198] In S1007, a job execution process is performed based on the setting values ​​of the job parameters. In the job execution process, the image forming apparatus 101 performs a series of image forming processes (e.g., copying, scanning, printing, etc.). Fig. 12A When the execution of a copy job is started when the "Auto (Color / B&W)" button is in a highlighted state in the screen shown in , the job is executed with the setting values ​​of "Auto (Color / B&W)" in this state as a job using these values.

[0199] In S1008 , as in S946 and S950 described above, the data sending and receiving unit 802 sends a job execution status notification to the server 102 .

[0200] Examples of interactions between the system and the user

[0201] exist 9A to 9C The processing shown in the sequence and Fig.10 and Fig.13 In the processing flow shown in FIG. 17A to FIG. 17D More specifically, examples of the voice control operation of the user 106 and the interaction given by the system in response to the voice control operation are illustrated. That is, Fig.17A 17D to 17D illustrate examples of utterances made by the user 106 as voice control operations and the corresponding on / off states of LEDs indicating voice processing states controlled by the voice control apparatus 100. Also shown are screens displayed on the operation panel 509 of the image forming apparatus and corresponding voices reproduced as responses by the voice control apparatus 100.

[0202] First, in the system, the LED is in the off state showing that the system is in a waiting state, and the top screen is displayed on the operation panel 509 (S1701). In this state, if the user 106 utters a wake-up word (S1702), the system starts to accept the voice control operation and turns on the LED to indicate that the voice processing is in progress (S1703). Note that S1702 is Fig.9A , and S1703 is a specific example of S906. After the wake-up word, if the user 106 utters "I want to make a copy" (S1704), the system flashes the LED to indicate that the response processing is in progress (S1705), and the system displays the copy top screen (S1706). Then, the system reproduces the response voice saying "Show copy screen" (S1707), and turns on the LED again to prompt the user 106 to make the next utterance (S1708). Note that S1704, S1705, S1706, S1707, and S1708 are respectively Fig.9ASpecific examples of S907, S908, S915, S919 and S920 are shown in FIG.

[0203] Next, if the user 106 issues "Document type" (S1709), the system flashes the LED (S1710) and displays the document type setting screen (S1711). Then, the system reproduces a response voice saying "Change of document type is allowed" (S1712), and turns on the LED (S1713). Note that S1709, S1710, S1711, S1712, and S1713 are respectively Fig.9A or Fig. 9B Specific examples of S921, S922, S915, S924 and S925 shown in FIG.

[0204] Next, when the user 106 issues "level adjustment" (S1714), the system flashes the LED (S1715) and displays the level adjustment setting screen (S1716). Then, the system reproduces a response voice saying "level adjustment change is allowed" (S1717), and turns on the LED (S1718). Note that S1714, S1715, S1716, S1717, and S1718 are respectively Fig.9A or Fig. 9B Specific examples of S921, S922, S915, S924 and S925 shown in FIG.

[0205] Next, when the user 106 issues a "select photo priority button three times" (S1719), the system flashes the LED (S1720) and moves the level adjustment (i.e., slider) to the right by three positions (S1721). Then, the system reproduces a response voice saying "level adjustment has changed" (S1722), and turns on the LED (S1723). This will be referred to later. Fig.13 Describes the details of the operation of the slider. Note that S1719, S1720, S1721, S1722, and S1723 are Fig.9A or Fig. 9B Specific examples of S926, S927, S915, S929 and S925 are shown in FIG.

[0206] Next, when the user 106 issues "OK" (S1724), the system flashes the LED (S1725), and displays the modified document type setting screen to reflect the level adjustment (S1726). Then, the system reproduces a response voice saying "The setting is confirmed" (S1727), and turns on the LED (S1728). Note that S1724, S1725, S1726, S1727, and S1728 are respectively Fig.9A or Fig. 9BSpecific examples of S931, S932, S915, S934 and S935 shown in FIG.

[0207] Next, when the user 106 issues "Copy start" (S1729), the system flashes the LED (S1730) and executes the copy job. When the execution of the copy job starts, the system displays a screen indicating that the copy job is being executed (S1732), and reproduces a response voice saying "Copy is starting" (S1731). When the execution of the copy job is completed, the system displays a screen indicating that the execution of the copy job is completed and reproduces a final response voice saying "Copy is completed" (S1733). In addition, the system turns off the LED to indicate that voice reception is disabled (S1734). Note that S1729, S1730, S1732, S1731, S1733, and S1734 are respectively Fig. 9C Specific examples of S936, S937, S944, S949, S954 and S955 are shown in FIG.

[0208] Voice control of sliders

[0209] According to the present embodiment, it is allowed to control the slider in various ways through the voice operation control in the above-mentioned S1719 to S1721 as an example. FIG. 22A to FIG. 22F : is a diagram illustrating a specific example of the operation of a slider by voice control.

[0210] Fig.22A An example of a manner in which the pointing position of the slider is moved from the center to the right by one position in response to the utterance "Move the slider to the right" is illustrated. Fig. 12A In the case of operating a screen similar to the screen shown in , the operation is controlled by voice including the button name, such as "Select the photo priority button once". In this embodiment, in addition to such normal voice operation control, screen-specific voice operation control (moving the slider according to voice indicating directions such as "right", "left", etc.) is also allowed.

[0211] In addition, on this screen, the same voice operation can be repeated multiple times. Fig. 22B Shows that the slider has been changed from Fig.22A In this embodiment, because the previous position of the slider is stored, the second utterance of "move the slider to the right" moves the slider to a position two positions away from the center position to the right.

[0212] You can distinguish between saying "Move the slider to the right" and saying "Move the slider to the right end". Fig.22D, the slider is moved to the right end (four positions away from the center) as shown in . In addition, voice operation control by utterances such as "move the slider to the center", "move the slider to the left end", etc. may be allowed. However, since the operation of the slider not based on the current position of the slider is not the subject of the present embodiment, its further detailed description is omitted.

[0213] As in Fig.22D In the case of , when the pointing position of the slider is at the right end of the sliding range, the slider cannot be moved further to the right. Fig.22D When voice operation control is performed in the state shown in to further move the slider to the right, it is determined that the slider cannot be moved, and the pointing position does not move. In this case, a temporary display (e.g., a pop-up display) may be performed to indicate that the requested movement is not allowed, and / or when Fig. 9B When reproducing the synthesized voice data in S929, the voice may tell that the requested slider movement is not allowed.

[0214] Fig. 22C An example of a manner in which the pointing position is moved two positions to the right from the center position in response to an utterance of "move the slider to the right twice" or "move the slider slightly to the right" is illustrated. As described above, by enabling the slider to be moved multiple times by one utterance, operability can be improved. Allowing ambiguous expressions such as "slightly" to be used in voice operation control enables an intuitive operation environment to be provided.

[0215] Fig.22E and Fig.22F Each example illustrates a way of moving the pointing position in response to the utterance "Move the slider to the right by about 1 / 4". By allowing the use of ratio expression in voice operation control as described above, it becomes possible to provide more flexible voice operation control. When ratio expression is used as described above, the amount of movement in units of scale changes depending on the current pointing position. Fig.22E In the example shown in , the current pointer position is in the center, and there are four remaining ruler notches to the right. Therefore, saying "Move the slider about 1 / 4" will move the slider one ruler notch to the right. Fig.22F In the example shown in , the current pointer position is at the left end, and there are eight remaining ruler scale marks to the right. Therefore, uttering "move the slider approximately 1 / 4 to the right" will move the slider two ruler scale marks to the right.

[0216] Processing flow of the voice control operation determination unit

[0217] Fig.13 1 is a flowchart illustrating an outline of the processing of the voice control operation determination unit 807 of the image forming apparatus 101 . Fig.13 The processing shown in corresponds to Fig.10S1003 and Fig.9A and Fig. 9C The processing in S914 and S943 in . FIG. 14A to FIG. 14E Illustrated in Fig.13 An example of a screen in which an operation to be performed by voice instruction is determined in the voice control operation determination process shown in FIG, and an example of a normal operation (by pressing a button or hardware key provided on the operation panel) and an operation controlled by voice operation. Fig.13 as well as FIG. 14A to FIG. 14E The processing flow of the voice control operation determination unit 807 will be described.

[0218] First, refer to FIG. 14A to FIG. 14E . Fig.14A An example of a screen (Chinese version) used in setting the priority assigned to text / photo in scanning in the copy level adjustment of the image forming apparatus 101 is illustrated. Fig.14E Shows the translation Fig.14A An example of the Japanese version screen of the Chinese version is shown in .

[0219] Buttons 1401 and 1402 are slider buttons for adjusting the degree of priority assigned to text and photo. When these buttons are operated, the slider provided in the center of the screen is moved. The current slider position is at the center of the slider bar, and moves to the left or right step by step when the right slider button or the left slider button is pressed. Button 1403 is a button for canceling the setting, and button 1404 is a confirmation button for reflecting the setting. Fig. 14B Shown with Fig.14A An example of screen control information related to the screen shown in . Fig. 14C and Fig.14D Shown with Fig.14A An example of voice control operation determination information related to the screen shown in . Fig. 14C and Fig.14D Rows 1405 to 1416 in represent row numbers of the voice control operation determination information.

[0220] Refer to the following Fig.14A The screen shown in FIG. 14B to FIG. 14D The example of the screen control information and the voice control operation determination information shown in FIG. Fig.13 The processing flow shown in .

[0221] In S1301, the voice control operation determination unit 807 Fig.10 The group ID determination result received in S1001 in the above embodiment is matched with the group ID information of the button included in the voice control operation determination information acquired in S1002. Fig.14A, if the group ID determination result is {ID: NUM00003, POS00016, FIN00011} (obtained when the user 106 issues "move the slider to the left three times"), then Fig. 14C and Fig.14D 00016+FIN00011" in row 1406 is matched, and the matching succeeds.

[0222] Note that Fig. 14C and Fig.14D In these examples shown in , assume that the user is viewing Fig.14E The Japanese version of the screen shown in the Fig.14E The voice control voice for the Japanese version of the screen shown in .

[0223] In S1302, the voice control operation determination unit 807 determines whether the matching in S1301 is successful. In the case of successful matching, the process proceeds to S1306, otherwise the process proceeds to S1303.

[0224] In S1303, the voice control operation determination unit 807 Fig.10 The group ID determination result received in S1001 in the example is matched with the group ID information of the hardware key included in the voice control operation determination information acquired in S1002. Fig.14A On the screen shown in , if the group ID determination result in the server 102 is {ID: OPR00011} (obtained when the user 106 issues "Start"), a match is made between the ID determination result and the group ID information of the hardware key described in the voice control operation determination information. In this case, the group ID "OPR00011" is hit in the voice control operation determination information, and the match is successful.

[0225] In S1304, the voice control operation determination unit 807 determines whether the matching in S1303 is successful. If the matching is successful, the process proceeds to S1305, otherwise the voice control operation determination process ends.

[0226] In S1305, the voice control operation determination unit 807 determines the hardware key hit in the matching process in S1303 by the voice control instruction operation, and the voice control operation determination unit 807 ends the voice control operation determination process. For example, when "start" with the group ID information "OPR00011" is hit, it is determined that the operation start key 207 is indicated, and the voice control operation determination process ends.

[0227] In S1306, the voice control operation determination unit 807 determines whether the operation target indicated in the matching result in S1301 is a slider button. For example, in the case of uttering "select text priority", in the case of Fig. 14C and Fig.14D In the matching process of rows 1405 to 1412 of the voice control operation determination information in the , row 1405 is hit. The "number" column in row 1405 has a value of 1401, so Fig. 14B In the screen control information, determine that the slider button is the operation object.

[0228] When it is determined that the operation target is a slider button, the pointing position of the slider to be operated is read. Fig.14A In the example shown in , the pointing position of the slider is at "0" (the value on the right side is positive and the value on the left side is negative). The movement of the slider on the scale is performed based on the read pointing position. In the case where the matched operation target is a slider button, the process proceeds to S1308, otherwise the process proceeds to S1307.

[0229] In S1307, the voice control operation determination unit 807 determines that the result of the matching in S1301 indicates that the voice operation control instruction operates the general button on the operation screen, and ends the voice control operation determination processing. For example, in the case of a match of "OK" with group ID "OPR00010", Fig.14D 1404 is described in the “number” column in row 1414 of the voice control operation determination information in , and therefore it is determined that the voice operation control instruction operation is the “OK” button as one of the general buttons.

[0230] In step S1308, the voice control operation determination unit 807 determines whether the group ID determination result includes a group ID that specifies the number of times the operation target is to be operated. Fig.10 In the case where the group ID determination result received in S1001 includes {ID: NUM00003} meaning "three times", it is determined that the group ID determination result includes the group ID of the specified number of times. The group ID indicating the number corresponds to Figure 7B 00001 to NUM99999 in the slider. However, the adjustment range of the slider is eight at most. Therefore, the group ID for the specified number of times is set in the range up to the maximum adjustment width of the slider (in this specific case, NUM00001 to NUM00008). If the group ID for the specified number of times is included, the process proceeds to S1310, otherwise the process proceeds to S1309.

[0231] In S1309, the voice control operation determination unit 807 determines that the result of the matching in S1301 specifies a single voice operation on the slider button on the operation screen, and ends the voice control operation determination processing. Fig. 14C In the case where row 1405 and row 1409 in are matched as group ID information, it is determined that the voice operation control indicates operation of the slider button once (ie, slider button 1401 or 1402 is to be operated once).

[0232] In S1310, the voice control operation determination unit 807 determines whether the group ID determined as the specified number of times in S1308 represents a specific value of the number of times. For example, if one of NUM00001 to NUM00008 is included in the group ID determination result, it is determined that the specific value of the number is specified. If the specific value is specified, the process proceeds to S1311, otherwise the process proceeds to S1312.

[0233] In S1311, the voice control operation determination unit 807 determines that the result of the matching in step S1301 indicates a voice control operation instructing to operate the slider button on the operation screen a specified number of times, and ends the voice control operation determination processing. Fig. 14C In the case where rows 1406 and 1410 are matched as group ID information (for example, row 1406 is matched when user 106 issues "select text priority twice"), it is determined that the voice operation control indicates the number of times the slider button is operated (that is, the number of times slider button 1401 or 1402 is operated).

[0234] In S1312, the voice control operation determination unit 807 determines whether the number of times is described in the group ID determined in S1308 as including the information specifying the number of times so that a special expression is used or calculation is required. The special numerical expression is, for example, "a small amount", "slightly", etc., and Fig.11D {DRG00018} in the group ID shown in is an example of such an expression. Special numerical expressions are not limited to these examples, and may include expressions that vaguely indicate an amount or number (such as "almost", "at most", "a little"). For example, in Fig.11H When a fractional expression such as "1 / 3" is used as in the group ID {FRA00103}, the number of times needs to be calculated.

[0235] In S1313, the voice control operation determination unit 807 determines that the result of the match in step S1301 indicates a voice control operation that instructs the slider button on the operation screen to be operated a specific number of times specified in the special expression, and ends the voice control operation determination process. For example, in the case where rows 1407 and 1411 in FIG. 14C are matched as group ID information, it is determined that the number of times specified by the voice control operation described in the group ID {DGR00018} is to be interpreted as the slider button being operated twice (i.e., the slider button 1401 or 1402 is pressed twice). Here, "slightly" associated with the group ID {DGR00018} is regarded as "twice". However, this value is not limited to "twice" but can be changed by making settings.

[0236] In S1314, the voice control operation determination unit 807 calculates the number of times. For example, in the case where the group ID information is {FRG00103, POS00012} (where "1 / 3" and "right" are specified), the number of moves is calculated based on the current position of the slider and the number of scale divisions remaining to the right. More specifically, in the case where Fig.14A In the example shown in , where the current position is at the center of the slider bar, the number of scale divisions remaining to the right is 4, and therefore 4×1 / 3=1.333≈1, which means that the number of moves is to be determined as 1. In this way, the number of moves is calculated based on the current position of the slider bar, the number of divisions remaining in the direction in which the bar is to be moved, and the fractional expression described in the group ID information.

[0237] In S1315, the voice control operation determination unit 807 determines that the result of the matching in step S1301 indicates that the voice control operation indicates operating the slider button on the operation screen as many times as the number calculated in S1314, and ends the voice control operation determination processing. Fig. 14C 0018} is matched as the group ID information, it is determined that the group ID {DGR00018} specifies "twice" as the number of times the operation is to be performed, and therefore it is determined that the voice control operation specifies pressing the slider button twice (i.e., the slider button 1401 or 1402 is to be pressed twice). Here, "slightly" associated with the group ID {DGR00018} is regarded as "twice". However, this value is not limited to "twice" but can be changed by making settings.

[0238] Replenish

[0239] As described above, according to the present embodiment, it is possible to provide an information processing system that allows a user to perform a voice control operation depending on the configuration of a setting screen of a device. In particular, it is possible to perform a voice control operation on a setting screen including a slider-shaped operation object. According to the present embodiment, it is possible to perform a voice control operation on a setting screen including a slider-shaped operation object without depending on the name of a button. In the present embodiment, a voice control operation can be performed to change the setting in consideration of the current setting state. In the present embodiment, the amount of movement of the slider can be specified by a numerical value (numerical information). In the present embodiment, the amount of movement of the slider can be specified by a ratio (ratio information). As described above, the operation can be flexibly controlled by voice, and thus excellent usability can be achieved.

[0240] Second embodiment

[0241] The first embodiment has been described above by way of example for a case where one slider-shaped operation object is arranged on a setting screen. In the second embodiment described below, the embodiment is described for a case where a plurality of slider-shaped operation objects are arranged on a setting screen.

[0242] Example of interaction between system and user

[0243] exist 9A to 9C The processing shown in the sequence and Fig.10 and Fig.15A and Fig. 15B In the processing flow shown in FIG. 17A to FIG. 17D More specifically, examples of the voice control operation of the user 106 and the interaction given by the system in response to the voice control operation are illustrated. That is, 18A to 18D An example of an utterance of the user 106 for a voice control operation and an on / off state of an LED controlled by the system for indicating a corresponding voice processing state of the voice control apparatus 100 are illustrated. A screen displayed on the operation panel 509 of the image forming apparatus and a corresponding voice reproduced as a response of the voice control apparatus 100 are also illustrated.

[0244] First, in the system, the LED is in an off state showing that the system is in a waiting state, and the top screen is displayed on the operation panel 509 (S1801). In this state, if the user 106 utters a wake-up word (S1802), the system starts to accept voice control operations and turns on the LED to indicate that the voice processing is in progress (S1803). Note that S1802 is a specific example of S905, and S1803 is a specific example of S905. Fig.9ASpecific example of S906 in . After the wake-up word, if the user 106 utters "I want to make a copy" (S1804), the system flashes the LED to indicate that the response processing is in progress (S1805), and the system displays the copy top screen (S1806). Then, the system reproduces the response voice saying "Show copy screen" (S1807), and turns on the LED again to prompt the user 106 to make the next utterance (S1808). Note that S1804, S1805, S1806, S1807, and S1808 are respectively Fig.9A Specific examples of S907, S908, S915, S919 and S920 are shown.

[0245] Next, if the user 106 issues "other functions" (S1809), the system flashes the LED (S1810) and displays the other function screen (S1811). Then, the system reproduces a response voice saying "other function 1 has been displayed" (S1812) and turns on the LED (S1813). Note that S1809, S1810, S1811, S1812, and S1813 are respectively Fig.9A or Fig. 9B Specific examples of S921, S922, S915, S924 and S925 are shown in FIG.

[0246] Next, when the user 106 issues "Next page" (S1814), the system flashes the LED (S1815) and displays the second other function screen (S1816). Then, the system reproduces a response voice saying "Other function 2 has been displayed" (S1817) and turns on the LED (S1818). Note that S1814, S1815, S1816, S1817, and S1818 are respectively Fig.9A or Fig. 9B Specific examples of S921, S922, S915, S924 and S925 are shown.

[0247] Next, when the user 106 issues "color adjustment" (S1919), the system flashes the LED (S1820) and displays the color adjustment setting screen (S1821). Then, the system reproduces a response voice saying "Allow color adjustment to be changed" (S1822) and turns on the LED (S1823). Note that S1819 is Fig.9A In the specific example of S921, S1820 is Fig.9A In the specific example of S922, S1821 is Fig.9A In the specific example of S915, S1822 is Fig. 9B A specific example of S924 in FIG. 1 and S1823 is Fig. 9B A specific example of S925 in FIG.

[0248] Next, when the user 106 issues "saturation adjustment" (S1824), the system flashes the LED (S1825) and displays the saturation adjustment setting screen (S1826). Then, the system reproduces a response voice saying "saturation adjustment change is allowed" (S1827), and turns on the LED (S1828). Note that S1824 is Fig.9A In the specific example of S921, S1825 is Fig.9A For example, S922 in S1826 is Fig.9A For example, S915 in S1827 is Fig. 9B In the specific example of S924, S1828 is Fig. 9B A specific example of S925 in FIG.

[0249] Next, when the user 106 utters "Yellow and cyan increase twice" (S1829), the system flashes the LED (S1830) and moves the saturation adjuster (i.e., the sliders associated with yellow and cyan, respectively) to the right twice (S1831). Then, the system reproduces a response sound saying "Saturation adjustment has been changed" (S1832), and turns on the LED (S1833). This will be referred to later. Fig.15A and Fig. 15B Describes the details of the operation of the slider. Note that S1829 is Fig. 9B For example, S926 in S1830 is Fig. 9B For example, S927 in S1831 is Fig.9A For example, S915 in Fig. 9B A specific example of S929 in FIG. 1 and S1833 is Fig. 9B A specific example of S925 in FIG.

[0250] Next, when the user 106 utters "Reduce cyan and magenta by three" (S1834), the system flashes the LED (S1835) and moves the saturation adjuster (i.e., the sliders for cyan and magenta, respectively) to the right by three positions (S1721). Then, the system reproduces a response voice saying "The saturation adjustment has been changed" (S1837), and turns on the LED (S1838). Note that S1834 is Fig. 9B For example, S926 in S1835 is Fig. 9B For example, S927 in S1836 is Fig.9A For example, S915 in S1837 is Fig. 9B For example, S929 in S1838 is Fig. 9B A specific example of S925 in FIG.

[0251] Next, when the user 106 issues "OK" (S1839), the system flashes the LED (S1840) and displays the modified saturation adjustment setting screen to reflect the saturation adjustment (S1841). Then, the system reproduces a response voice saying "The setting has been fixed" (S1842) and turns on the LED (S1843). S1839 is Fig. 9B In the specific example of S931, S1840 is Fig. 9B In the specific example of S932, S1841 is Fig.9A For example, S915 in S1842 is Fig. 9B A specific example of S934 in FIG. 1 and S1843 is Fig. 9C A specific example of S935 in FIG.

[0252] Next, when the user 106 issues "Copy start" (S1844), the system flashes the LED (S1845) and executes the copy job. When the copy job starts to be executed, the system displays a screen indicating that the copy job is being executed (S1846), and reproduces a response voice saying "Copy is starting" (S1847). When the copy job is completed, the system displays a screen indicating that the copy job is completed, and reproduces a final response voice saying "Copy is completed" (S1848). In addition, the system turns off the LED to indicate that voice reception is disabled (S1849). Note that S1844, S1845, S1846, S1847, S1848, and S1849 are respectively Fig. 9C Specific examples of S936, S937, S944, S949, S954 and S955 are shown.

[0253] Example of operation with multiple sliders

[0254] In the present embodiment, the voice control of the slider illustrated in S1829 to S1831 and S1834 to S1836 may be performed in various ways. FIG. 23A to FIG. 23D is a diagram illustrating a specific example of the operation of a slider controlled by voice.

[0255] Fig.23A An example of a manner in which the pointing position of the yellow slider is moved one position from the center to the right in response to uttering "increase yellow" is illustrated. In the present embodiment, in the case where the setting screen includes a plurality of sliders, a specific slider can be specified by using a name (a color in this example) that identifies the slider, so that only a specific slider of the plurality of sliders can be moved. To specify the yellow slider, both uttering "yellow" and uttering "yellow slider" are permissible. It is permissible to utter "right" while not permissible to utter "high".

[0256] Fig. 23BThe example shows a way of moving the sliders by issuing "Increase all colors" to move all sliders on the screen (sliders of yellow, green, cyan, blue, magenta and red) one position to the right from their current pointing positions. As described above, this embodiment provides excellent operability when operating multiple sliders at the same time.

[0257] Fig.23C The manner of moving the sliders is illustrated in which the yellow and cyan sliders are moved two positions to the right from their current pointing positions by issuing "yellow and cyan increase twice." As described above, according to the present embodiment, two or more sliders among a plurality of sliders can be operated at once.

[0258] Fig.23D The manner of moving the sliders is illustrated in which the sliders of cyan and magenta are moved three positions to the left from their current pointing positions by issuing "reduce cyan and magenta by three". That is, the slider of cyan is moved from the current position, that is, two scales to the right from the center position, to the left scale directly adjacent to the center position, and the slider of magenta is moved three scales to the left from the center position. As described above, according to the present embodiment, two or more sliders among a plurality of sliders can be moved while taking into account the current positions of the plurality of sliders.

[0259] Processing flow of the voice control operation determination unit when there are multiple sliders

[0260] Fig.15A and Fig. 15B 2 is a flowchart illustrating an outline of processing of the voice control operation determination unit 807 of the image forming apparatus 101 in the case where the image forming apparatus 101 includes a plurality of sliders. Fig.15A and Fig. 15B The processing shown corresponds to Fig.10 S1003 in Fig.9A The S915 and Fig. 9C S943 shown in FIG. FIG. 16A to FIG. 16E An example of a screen in which a plurality of sliders to be operated by voice control operation are operated is illustrated. Fig.15A The operation on the slider is determined by the voice control operation determination process shown in FIG. 15B .

[0261] Refer to the following Fig.15A and Fig. 15B as well as FIG. 16A to FIG. 16E , the processing flow of the voice control operation determination unit 807 is described for the case where there are multiple sliders on the screen.

[0262] First, it explains FIG. 16A to FIG. 16E . Fig.16A An example of a screen (Chinese version) for adjusting the saturation of the image forming apparatus 101 is illustrated, and Fig.16EThe screen for adjusting the saturation of the image forming apparatus 101 is illustrated (corresponding to Fig.16A 16. The Japanese version of the Chinese version shown in FIG. 16 is an example of a Japanese version of the Chinese version shown in FIG. 16. Buttons 1601 to 1614 are slider buttons for adjusting the saturation level, that is, buttons for operating corresponding to the slider located in the center of the screen. Button 1615 is a button for canceling the setting, and button 1616 is a confirmation button for reflecting the setting. Fig. 16B Illustrated Fig.16A An example of screen control information on the screen shown. Fig. 16C and Fig.16D Illustrated with Fig.16A An example of voice control operation determination information associated with the screen shown. Fig. 16C and Fig.16D Lines 1617 to 1634 in the table represent the lines of voice control operation confirmation information. Fig. 16C and Fig.16D In the examples shown in , assume that the user is viewing Fig.16E The Japanese version of the screen is shown, and Fig.16E The Japanese version screen shown emits voice for voice-operated control.

[0263] Reference below Fig.15A and Fig. 15B The processing flow shown, Fig.16A The screen shown in Figure 16B to Fig. 16C The processing is described by the screen control information shown. S1501 to S1515 are similar to Fig.13 S1301 to S1315 are shown, therefore, the following description will focus on the different processes in S1516 to S1518.

[0264] In S1516, based on Fig.10 The data obtained in S1002 is Fig. 16B The screen control information in the image is used to determine whether there are two or more sliders. Fig. 16B In the case where there are two or more sliders, the process proceeds to S1517. However, when the number of sliders is one or less, the process proceeds to S1501.

[0265] In S1517, it is determined that Fig.10 The group ID received in S1001 determines whether the result includes two or more slider names. The slider name is, for example, Fig.16AIn the case where the group ID determination result is {COL0004, COL0006, NUM00002, DRG00013} (obtained when the user issues "increase yellow and cyan by two"), it is determined from the group ID {COL0004, COL0006} that two or more slider names "yellow" and "cyan" are included. Fig.16A When there are two or more slider names as in the illustrated case, the process proceeds to S1518, otherwise the process proceeds to S1501.

[0266] In S1518, the group ID determination result is reproduced for each slider name determined in S1517. For example, in the case where the group ID determination result is {COL0004, COL0006, NUM00002, DRG00013} (as in the example in S1517), two group ID determination results are reproduced for each slider name, such as {COL0004, NUM00002, DRG00013} and {COL0006, NUM00002, DRG00013}. By reproducing the group ID determination results in the above manner, the user can simultaneously operate two or more sliders by making a single utterance.

[0267] A brief description of the processing flow after S1501 is given below for the case where the group ID determination result is {COL0004, COL0006, NUM00002, DRG00013} (obtained when the user issues "increase yellow and cyan by two").

[0268] In S1518, two group ID determination results {COL0004, NUM00002, DRG00013} and {COL0006, NUM00002, DRG00013} are reproduced. In S1501, these group ID determination results are respectively Fig. 16C The group ID information in the voice control operation determination information matches.

[0269] Next, it is determined in step S1502 that the match is successful, and the process further proceeds to S1506. More specifically, the two group ID determination results are respectively Fig. 16C Line 1620 and line 1624 in match.

[0270] In S1506, it is determined that the two group ID determination results each represent a slider button (ie, Fig. 16B 1604 and 1608), and therefore the processing proceeds to S1508.

[0271] In step S1508, it is determined that the respective group ID determination results include the group ID {NUM0002} the specified number of times, and thus the process proceeds to S1510.

[0272] In S1510 , the determination number is represented by a specific value (ie, “2”), and the process proceeds to S1511 .

[0273] In S1511, it is determined that the voice control operation specifies that each slider button on the screen corresponding to the group ID information is operated a specified number of times. More specifically, Fig. 16C The row 1620 and the row 1624 in are matched as the group ID information. Therefore, it is determined that the voice control operation specifies to operate each of the yellow and cyan sliders twice (ie, to press each of the slider buttons 1604 and 1608 twice).

[0274] By performing the processing as described above, when the user issues "increase yellow and cyan by two", two or more sliders can be operated at the same time.

[0275] As described above, according to the present embodiment, a voice control operation can be performed on a setting screen including a plurality of slider-shaped operation objects. In the present embodiment, on a setting screen having a plurality of sliders, a voice control operation can be performed so that the pointing scale is changed only for a specific slider. In the present embodiment, a plurality of sliders can be moved simultaneously. In the present embodiment, the movement amount of a plurality of sliders can be specified by using values. In the present embodiment, a voice control operation can be performed to change the setting in consideration of the current setting state. As described above, the operation can be flexibly controlled by voice, so excellent usability can be achieved. Note that in the above reference Fig. 16C In the example of determining information about the voice control operation related to controlling multiple sliders described above, the example of omitting information represented by a special expression and omitting it is the same as the above reference Fig.13 S1309, S1313 and S1315 and Fig. 14C The example of the number of times the operation is performed is the same as the case described in the group ID information. However, it can be processed in a similar manner to control a plurality of sliders.

[0276] Third embodiment

[0277] The third embodiment described below discloses a method for changing a setting value by a voice control operation so that a specific selection object among a plurality of selection objects is selected and the setting value is changed in this state. In the above-mentioned first and second embodiments, the voice control operation is explained with respect to the case where the result of the voice control operation determination processing in S1003 indicates that the voice control operation specifies transition to a slider screen or movement of a slider.

[0278] That is, in the third embodiment, based on Fig.27BThe screen control information and Fig.27C The focus position (focus position) is changed based on the voice control operation determination information in the process, thereby updating the displayed screen. In this process, the focus position is changed based on the current focus position read at S11306 described later. After the setting of the focus is completed, the setting parameters of the data management unit 803 of the device control program 801 are updated. The button set at the focus position is called a focus button (focus object), and the object used to highlight the focus object is called a highlighted object.

[0279] Note that the third embodiment is similar to the first embodiment except for the characteristic parts. Therefore, similar parts are denoted by similar reference numerals, and further description thereof will be omitted.

[0280] FIG. 28A to FIG. 28D 9A to 9B are schematic diagrams illustrating an example of interaction between a voice control operation performed by user 106 and the system in response to the voice control operation. Fig. 9C The examples described in the sequence shown and Fig.26 That is, FIG. 28A to FIG. Fig.28D An example of an utterance given by the user 106 as a voice control operation corresponding to the corresponding voice processing performed by the voice control device 100 of the system and the on / off state of the LED indicating the voice processing state is illustrated. Also shown is a screen displayed on the operation panel 509 of the image forming apparatus and a corresponding voice reproduced as a response by the voice control device 100.

[0281] First, in the system, the LED is in the off state indicating that the system is in the voice waiting state, and the top screen is displayed on the operation panel 509 (S11501). In this state, if the user 106 issues a wake-up word (S11502), the system starts to accept voice control operations and turns on the LED to indicate that voice processing is in progress (S11503). Note that S11502 is Fig. 9A In the specific example of S905, S11503 is Fig. 9A Specific example of S906 in . After the wake-up word, if the user 106 utters "I want to make a copy" (S11504), the system flashes the LED to indicate that the response processing is in progress (S11505), and the system displays the copy top screen (S11506). Then, the system reproduces the response voice saying "Show copy screen" (S11507), and turns on the LED again to prompt the user 106 to utter the next voice (S11508). Note that S11504 is Fig. 9A In the specific example of S907, S11505 is Fig. 9A9A , S11506 is a specific example of S915, and S11507 is a specific example of S908 in FIG. Fig. 9A In the specific example of S919, S11508 is Fig. 9A A specific example of S920 in FIG.

[0282] Next, if the user 106 issues "Move focus to the right" (S11509), the system flashes the LED (S11510) and displays the copy top screen (settings screen) that has been modified to move the focus to the right (S11511). Then, the system reproduces a response voice saying "Focus has been moved" (S11512), and turns on the LED (S11513). Note that S11509 is Fig. 9B In the specific example of S926, S11510 is Fig. 9B S927, S11511 are Fig. 9A S915, S11512 are Fig. 9B S929, S11513 are Fig. 9B A specific example of S930 in FIG.

[0283] Next, when the user 106 issues "Focus moved down three" (S11514), the system flashes the LED (S11515) and displays the copy top screen (settings screen) that has been modified so that the focus is moved down three (S11516). Then, the system reproduces a response voice saying "Focus moved" (S11517), and turns on the LED (S11518). Note that S11514 is Fig. 9B For example, S926 in S11515 is Fig. 9B For example, S927 in S11516 is Fig. 9A In the specific example of S915, S11517 is Fig. 9B 9B is a specific example of S929, and S11518 is a specific example of S930 in FIG. 9B.

[0284] Next, when the user 106 issues "Select the focus button" (S11519), the system flashes the LED (S11520) and displays the copy top screen that has been modified so that the focus button is selected. Then, the system reproduces a response voice saying "The focus has been moved" (S11522) and turns on the LED (S11523). This will be referred to later. Fig.26 Describes the details of the focusing operation. Note that S11519 is a specific example of S931 in FIG. 9B, and S11520 is Fig. 9B In the specific example of S932, S11521 is Fig. 9A In the specific example of S915, S11522 is Fig. 9BIn the specific example of S934, S11523 is Fig. 9C A specific example of S935 in FIG.

[0285] Next, when the user 106 issues "Copy start" (S11524), the system flashes the LED (S11525) and executes the copy job. When the copy job starts to be executed, the system displays a screen indicating that the copy job is being executed (S11527), and reproduces a response voice saying "Copy is starting" (S11526). When the execution of the copy job is completed, the system displays a screen indicating that the copy job execution is ended, and reproduces a final response voice saying "Copy completed" (S11528). In addition, the system turns off the LED to indicate that voice reception is disabled (S11529). Note that S11524, S11525, S11527, S11526, S11528, and S11529 are respectively Fig. 9C Specific examples of S936, S937, S944, S949, S954 and S955 are shown.

[0286] Processing flow of the voice control operation determination unit

[0287] Fig.26 1 is a flowchart illustrating an outline of processing by the voice control operation determination unit 807 of the image forming apparatus 101 . Fig.26 The processing shown corresponds to Fig.10 S1003 in Fig. 9A S914 and Fig. 9C S943 in. FIG. 27A to FIG. 27E Illustrated in Fig.26 The illustrated example shows an example of a screen for determining an operation to be performed by voice in the voice control operation determination process, and examples of a normal operation (by pressing a button provided on the operation panel or a hardware key) and an operation controlled by voice operation.

[0288] Refer to the following Fig.26 and FIG. 27A to FIG. 27E The processing flow of the voice control operation determination unit 807 is described.

[0289] First, refer to FIG. 27A to FIG. 27E .

[0290] Fig.27A An example of a screen (Chinese version) for adjusting the saturation of the image forming apparatus 101 is illustrated. Fig.27E The screen for adjusting the saturation of the image forming apparatus 101 is illustrated (corresponding to Fig.27AThe focus 11401 is a highlighted object having a frame shape displayed as being superimposed on a specific button, thereby indicating that the button is in a selected state. Buttons 11402 to 11406 are buttons for changing various copy settings. Fig.27A , the focus 11401 is displayed in an overlay manner on the frame of the 100% button 11402. In this embodiment, when the screen is switched according to the voice control operation, the focus is displayed on the switched screen (copy top screen). Note that the focus can always be displayed regardless of whether the screen switching is performed by the voice control operation. Alternatively, Fig. 30A As shown, in response to the user's utterance "Show focus", the focus is displayed at the default position, and then the focus can be moved to another button (e.g., a text / photo button) represented by an icon of two keys positioned adjacent to the default position on the right or below the default position. Fig. 30A 30B is a diagram illustrating the way in which the focus displayed at the default position moves. The timing of displaying the focus can be changed by setting. The default position of the focus is the leftmost button among multiple buttons (selection objects) on the screen. Please note that the default focus position may be on any button in the lower left corner, upper right corner, lower right corner, or near the center. For example, the default position can be switched by setting. Alternatively, the focus can be placed on a button specified by the user through the button name, and the focus operation can start from this state. For example, as shown in FIG. 30B , the focus can first be placed on the concentration button 11406, and then the focus can be moved to other buttons, such as the button represented by the icon located directly below (text / photo button). FIG. 30B is a diagram illustrating the way in which the focus displayed at a position specified by the user is moved.

[0291] In the present embodiment, for the convenience of the description given later, the operation performed after the focus has moved to the 100% button 11402 is first described. Fig.27B Illustrated in Fig.27A An example of a screen obtained after the position of the focus 11401 in FIG. 1 is moved to the position of the concentration button 11406 . Fig.27A and Fig.27B The screen shown in includes buttons for changing settings for copying in addition to buttons 11402 to 11406, but the following description focuses on buttons 11402 to 11406 for simplicity. Fig.27C Illustrated Fig.27A and Fig.27B An example of screen control information on the screen shown. Fig.27D Illustrated Fig.27A and Fig.27B An example of voice control operation determination information on the screen shown. Fig.27D11407 to 11416 in the figure represent rows of voice control operation determination information.

[0292] Reference Fig.27A The screen shown and Fig.27B and Fig.27C The example of the screen control information and the voice control operation confirmation information shown in FIG. Fig.26 Processing shown.

[0293] In S11301, Fig.10 The group ID determination result received in S1001 of FIG. 1 is matched with the group ID information of the buttons included in the voice control operation determination information acquired in S1002, excluding the hardware keys (rows 11415 and 11416). For example, in the display Fig.27A In the case of the screen shown, if the group ID determination result is {ID: UIP00012, POS00016, FIN00011}, then Fig.27D The group ID information in the column "Group ID information related to the display screen" in row 11407 of the voice control operation determination information in the 11407 is matched with the group ID determination result. In this case, the group ID "UIP00012+POS00016+FIN00011" in row 11407 is hit, and the match is successful. Note that UIP00012 is information indicating the focus, POS00016 is information indicating the direction, and +FIN00011 is information indicating movement. Note that POS00016 is direction information indicating the direction.

[0294] Please note, Fig.27D The examples shown in the Fig.27E Voice control operation of the Japanese version of the screen shown in .

[0295] In S11302, it is determined whether the matching in S11301 is successful. In the case of successful matching, the process proceeds to S11306, otherwise, the process proceeds to S11303.

[0296] In S11303, Fig.10 The group ID determination result received in S1001 in the above embodiment is matched with the group ID information for the hardware key included in the voice control operation determination information acquired in S1002. Fig.27AOn the screen shown, if the group ID determination result given by the server 102 is {ID: OPR00011} (which is obtained when the user 106 issues "start"), a match is made between the group ID determination result and the group ID information for the hardware key in the voice control operation determination information. In this case, the group ID "OPR00011" is hit in the voice control operation determination information, and the match is successful.

[0297] In S11304, it is determined whether the matching in S11303 is successful. If the matching is successful, the process proceeds to S11305, otherwise the voice control operation determination process is terminated.

[0298] In S11305, it is determined that the hardware key matched in S11303 is the operation target of the voice control operation, and the voice control operation determination processing is terminated. For example, when "start" with group ID information "OPR00011" is hit, it is determined that the voice control operation specifies the operation start key 207, and the voice control operation determination processing is terminated.

[0299] In S11306, in the matching result of S11301, it is determined whether the focus 11401 is designated as the operation target. Fig.27D In the case where rows 11407 to 11409 of the voice control operation determination information in the embodiment match (for example, when the user 106 utters "move focus left", row 11407 matches), 11401 is described in the "number" column in row 11401, therefore, according to Fig.27A or Fig.27B The screen control information in determines that the focus 11401 is specified by the voice control operation. Note that if it is determined that the focus is specified, the current position of the focus specified to be operated is read. In the case where the matched operation target is the focus 11401, the process proceeds to S11308, otherwise the process proceeds to S11307.

[0300] In S11307, it is determined that the matching result in S11301 indicates that the voice control operation specifies the operation of the button on the operation screen, and the voice control operation determination processing is ended. For example, in the case of matching with "reduction" with group ID "DGR0003", Fig.27D In the "number" column of row 11408 of the voice control operation determination information shown, 11404 is described, thereby determining that the voice control operation specifies the operation of the "zoom out" button according to the screen control information of FIG. 27D.

[0301] In S11308, it is determined whether the group ID determination result includes a group ID for a specified direction. For example, {ID: POS00016} indicating "left" is included in Fig.10In the case of the group ID determination result received in S1001, the group ID of the specified direction is included in the group ID determination result. The group ID indicating the direction corresponds to Figure 25G POS00010 to POS00017 in , however, for simplicity of explanation, oblique directions such as upper right, lower left, etc. are not included. In the case of a group ID including a designated direction, the process proceeds to S11310, otherwise the process proceeds to S11309.

[0302] In S11309, it is determined whether the group ID determination result includes the group ID of the specified number of times. Fig.10 In the case where the group ID determination result received in S1001 includes {ID: NUM00002} (numeric information) indicating "2", it is determined that the group ID determination result includes the group ID of the specified number of times. Figure 7B , the group ID indicating the number corresponds to NUM00001 to NUM99999. However, since the maximum movement of the focus is 8, the group ID of the specified number of times is within the maximum allowable range of the focus movement (NUM00001 to NUM00008). If the group ID of the specified number of times is included, the process proceeds to S11311, otherwise the process proceeds to S11310.

[0303] In S11310, it is determined that the matching result in S11301 indicates that the voice control operation specifies to move the focus 11401 to the next button on the operation screen, and the voice control operation determination processing is ended. Fig.27D , it is determined that the voice control operation specifies moving the focus 11401 to an adjacent button (i.e., moving the focus 11401 to one of the buttons located above, below, to the left, or to the right of the current focus position). Fig.27A In the case where the group ID determination result is {ID: UIP00012, POS00016, FIN00011}, it is determined that the voice control operation specifies that the focus 11401 is moved to the left by a button based on {POS00016} indicating "left". Fig.10When the screen update process is performed in step S1004 of , the screen is updated so that the focus 11401 moves from the 100% button 11402 to the concentration button 11406, and is displayed as superimposed on the concentration button 11406. In this way, the user 106 is allowed to move the display position of the focus 11401 to one of the button positions above, below, to the left, and to the right of the current button. The buttons above, below, to the left, and to the right of the focus 11401 are preset. That is, in this example, the predetermined allowable movement destinations are the minus button 11403, the zoom out button 11404, the concentration button 11406, and the manual feed button 11405 located above, below, to the left, and to the right of the current focus position of the 100% button 11401.

[0304] In S11311, it is determined that the matching result in S11301 indicates that the voice control operation specifies moving the focus 11401 by operating two or more buttons on the screen, and the voice control operation determination processing is ended. Fig.27A In the case of the screen in FIG. 1 , if the group ID determination result is {ID: UIP00012, NUM00002, POS00014} (for example, obtained when the user 106 says "move the focus down two buttons"), {NUM0002} means "2" and {POS00014} means "down", so it is determined that the voice control operation specifies moving the focus 11401 down two buttons. Fig.10 When the screen update process is performed in step S1004, the screen is updated so that the focus 11401 moves from the 100% button 11402 to the auto zoom button and is displayed as superimposed on the auto zoom button. In this way, the user 106 is allowed to move the display position of the focus 11401 by two or more positions at a time to one of the buttons located above, below, to the left, or to the right of the current focus position.

[0305] In S11312, the button on the operation screen where the voice control operation specifies the operation focus 11401 is located is determined, and the voice control operation determination processing is ended. Fig.27D When row 11409 in is matched as group ID information, it is determined that the voice control operation specifies the button where the current focus 11401 is located. For example, when the display Fig.27A When the screen is displayed, if the group ID determination result is {ID: UIP00012, OPR00040}, {OPR00040} means "select", so it is determined that the voice control operation specifies the selection of the button where the current focus 11401 is located. Then, in Fig.10 In S1004 in , the screen is updated so that the screen is displayed in the same manner as when the 100% button 11402 is pressed.

[0306] As described above, user 106 is allowed to operate the position of focus 11401 by voice. Therefore, even if there is a button to which no name is assigned, the button can be selected by moving the position of focus 11401 to the button (in the same way as when pressing the operation panel with a finger).

[0307] Examples of how focus is displayed

[0308] The focus position can be displayed in various ways according to the screen configuration, visibility, etc. Fig.29A An example is illustrated where the button in focus is surrounded by a dark frame, clearly informing the user of the focus location. Fig.29B An example is illustrated in which the focus is clearly indicated to the user 106 by pointing the focus button with an arrow (pointing icon). Fig.29C An example is illustrated in which the entire focused button is displayed in a dark and conspicuous color, and the text is displayed in an opposite color, thereby clearly notifying the user 106 that the button is focused.

[0309] like FIG. 29A to FIG. 29C As shown, various methods can be used to indicate the focus position. However, preferably, a button in a selected state, a button in a non-selected state, and a button in a focused state are distinguished (identifiable). The method of displaying the focus can be switched by setting.

[0310] Replenish

[0311] As described above, according to the present embodiment, the user 106 can operate the operation panel 509 of the image forming device 101 by voice. Specifically, according to the present embodiment, the focus position on the screen can be controlled so that the focus moves from the current focus position to one of the positions above, below, on the left, and on the right according to the utterance made by the user. In addition, the button where the focus is located can be operated by controlling the selection focus with the help of voice operation. Therefore, by controlling the focus, even a selection object whose name is unknown can be specified. According to the present embodiment, an operation of selecting a selection object (changing the selection state of the selection object) and an operation of placing focus on the selection object (changing the focus state) can be selectively performed. According to the present embodiment, the conditions for starting the focus display can be switched. According to the present embodiment, the method of indicating the focus can be switched.

[0312] Other embodiments

[0313] The present disclosure is not limited to the above-described embodiments, and various modifications (including organic combinations of the embodiments) can be made based on the spirit of the present disclosure. Note that any such modifications also fall within the scope of the present disclosure. That is, any modification of any embodiment and any combination including the embodiments falls within the scope of the present disclosure.

[0314] The present disclosure may be applied to a system including a plurality of devices, or to a device including a single device. In the above-described embodiment, the voice control device 100, the image forming device 101, and the server 102 respectively execute the voice control program 601, the device control program 801, and the voice data conversion control program 701. However, all programs may be executed by the image forming device 101 having a microphone. The server 102 may be separated, and the voice control program 601 and the device control program 801 may be executed by the image forming device 101.

[0315] In the above-described embodiment, the operation on the operation screen has been described above for a part of many operation screens provided by the image forming apparatus 101. Note that the method of controlling a specific screen by natural language can be applied to other screens. In particular, the voice control operation on the screen including the slider-shaped operation object or the voice control operation using the focus can be applied to other screens.

[0316] FIG. 19A to FIG. 19C , FIG. 20A to FIG. 20C as well as FIG. 21A to FIG. 21C 101 are diagrams showing a list of setting items in an image forming apparatus. In the table of the list shown in these diagrams, function classes are described in the first column (i.e., the leftmost column). Functions provided in the image forming apparatus 101 as an MFP include a printing function, a copying function, and a sending function. On the image forming apparatus 101, in addition to settings related to these functions, settings can also be made for the main body parts necessary for the operation of the apparatus. FIG. 19A to FIG. 19C , FIG. 20A to FIG. 20C as well as Fig.21A In the table of the list shown in FIG. 21C , the item group is described in the second column as counting from the leftmost column. An item group is a group of items that are grouped together by similar categories in the items operated using a function. An item group is output as one or more screens. FIG. 19A to FIG. 19C , FIG. 20A to FIG. 20C as well as FIG. 21A to FIG. 21C In the table of the list shown, the items are described in the third column as counting from the leftmost column. These items are operation items or setting items that are allowed to be operated on the screen displayed for each item group. Items common in the function column or item group column are operation items that can be used on almost all screens except for some special screens. In the present disclosure, not only can the settings be changed on the screens described above with reference to the embodiments for level adjustment, saturation adjustment, etc., but the present disclosure can also be applied to changing FIG. 19A to FIG. 19C , FIG. 20A to FIG. 20C as well as FIG. 21A to FIG. 21C In addition to the above settings, the present disclosure can also be applied to sliding in the display area on the page through voice control operations such as "move two pages to the right", "scroll the page about 1 / 2", etc. by way of example.

[0317] The embodiments of the present invention may also be implemented by reading out and executing computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more completely referred to as a "non-transitory computer-readable storage medium") to perform one or more functions in the above-mentioned embodiments, and / or a computer of a system or device including one or more circuits (e.g., an application-specific integrated circuit (ASIC)) for performing one or more functions in the above-mentioned embodiments, and the embodiments of the present invention may be implemented by a method in which, for example, a computer of a system or device reads out and executes computer executable instructions from a storage medium to perform one or more functions in the above-mentioned embodiments, and / or controls one or more circuits to perform one or more functions in the above-mentioned embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)), and may include a network of separate computers or separate processors to read out and execute computer executable instructions. Computer executable instructions may be provided to the computer, for example, from a network or a storage medium. The storage medium may include, for example, a hard disk, a random access memory (RAM), a read-only memory (ROM), a memory of a distributed computing system, an optical disk (such as a compact disc (CD), a digital versatile disc (DVD), or a Blu-ray disc (BD)) TM ), a flash memory device, a memory card, and the like.

[0318] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0319] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An information processing system, comprising: a display capable of displaying information; A microphone capable of picking up sound; as well as One or more controllers configured to: a unit configured to control to display a screen on the display, wherein the screen includes a plurality of selection objects, wherein on the screen, one of the plurality of selection objects is highlighted to indicate that the one of the plurality of selection objects is a focus target; a unit configured to acquire first identification information and second identification information based on voice information acquired via the microphone, the first identification information corresponding to the target of interest, and the second identification information related to a location update process of the target of interest; a unit configured to determine a current position of a current target of interest corresponding to the first identification information; a unit configured to determine another selection object to be highlighted as a focus target based on the determined current position and the second identification information; and A unit configured to perform control to display on the display the screen in which the focus target has been updated by stopping highlighting the one selection object and highlighting the other selection object.

2. The information processing system according to claim 1, wherein: obtaining single-phrase text information from speech information using at least a learning model learned using training data, the training data comprising pairs of text information and speech information, and The first identification information and the second identification information are acquired from the single phrase text information.

3. The information processing system according to claim 1, wherein: The identification information acquired from the voice information includes digital information for updating the pointing position of the attention target.

4. The information processing system according to claim 1, wherein: The second identification information acquired from the voice information includes direction information for determining the other selection object.

5. The information processing system according to claim 1, wherein: The screen is a screen for setting the copy function.

6. The information processing system according to claim 1, wherein: The screen is a screen for setting the sending function.

7. The information processing system according to claim 1, wherein: The screen is a screen for setting a print function.

8. The information processing system according to claim 1, wherein: The screen is a screen for setting the main body part.

9. The information processing system according to claim 1, further comprising: a speaker configured to output sound; as well as A unit configured to provide a notification via the speaker in response to an update.

10. The information processing system according to claim 9, further comprising: A unit configured to generate synthesized speech for use in the notification.

11. The information processing system according to claim 1, wherein: The plurality of selection objects are respectively parameters related to settings in forming an image by the image forming apparatus.

12. An information processing method, comprising: preparing a display capable of displaying information; Prepare a microphone that can pick up sound; Controlling to display the following screen on the display: the screen includes a plurality of selection objects, wherein on the screen, one of the plurality of selection objects is highlighted to indicate that the selection object in the plurality of selection objects is a focus target; Acquire first identification information and second identification information based on voice information acquired via the microphone, wherein the first identification information corresponds to the target of interest, and the second identification information is related to a location update process of the target of interest; Determine a current position of a current target of interest corresponding to the first identification information; determining another selection object to be highlighted as a focus target based on the determined current position and the second identification information; and Control is performed to display the screen on which the focus target has been updated by stopping highlighting the one selection object and highlighting the other selection object on the display.

Images