Information processing device, information processing system, control method, and program
The system automatically adjusts smart speaker settings based on user identification and language preferences, addressing the inefficiency of manual configuration in multi-user scenarios, thereby improving user experience.
Patent Information
- Application Number
- JP2020106388
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-06-19
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-06-19
AI Technical Summary
Existing smart speakers require manual adjustment of settings for each user, which is cumbersome and inefficient when used by multiple users in an office environment.
An information processing system that includes a multifunction peripheral (MFP), a smart speaker, and a cloud server, which automatically adjusts audio settings based on user identification, language, and country information, allowing seamless integration and personalized user experiences.
Facilitates easy and efficient configuration of smart speaker audio settings for different users, enhancing usability and convenience in multi-user environments.
Smart Images

Figure 0007580950000001 
Figure 0007580950000002 
Figure 0007580950000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an information processing device, an information processing system, a control method, and a program, and in particular to an information processing device that cooperates with a smart speaker that allows voice operation, and an information processing system including these, a control method, and a program. [Background technology]
[0002] Office equipment such as printers that work in conjunction with smart speakers is becoming more common (see, for example, Patent Document 1).
[0003] To use such a smart speaker comfortably, it is desirable to appropriately adjust the settings of the smart speaker. On the other hand, when the office equipment linked to the smart speaker is placed in an office, it is used by various users. Therefore, the comfortable operating environment of the smart speaker varies from person to person depending on the user who uses the linking function. For example, the language in which the user speaks to the smart speaker and the language that the smart speaker should use to speak to the user vary from user to user. Therefore, it is desirable to appropriately set the smart speaker for each user. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2019-18394 A Summary of the Invention [Problem to be solved by the invention]
[0005] However, it is time-consuming to change the settings at the user's instruction every time a different user starts using the smart speaker.
[0006] Therefore, the object of the present invention is to The voice settings of the smart speaker can be set to be easy for the user to hear.The present invention provides an information processing device, an information processing system, a control method, and a program. [Means for solving the problem]
[0007] The information processing device according to claim 1 of the present invention is an information processing device that cooperates with a smart speaker, and includes a first acquisition means for acquiring user identification information and a language information And country information, A second acquisition means for acquiring the language information a specifying means for specifying one language from the list of languages, and specifying the language corresponding to the country information from among the specified languages; and setting a voice setting of the smart speaker to the language corresponding to the country information specified by the specifying means. For the smart speaker Need and a requesting means for requesting the same.
[0008] Claims of the invention 5 The information processing system according to the present invention is an information processing system in which an information processing device, a smart speaker, and a server that manages settings of the smart speaker cooperate with each other, and the information processing device includes a first acquisition means for acquiring identification information of a user, and a language information And country information, A second acquisition means for acquiring the language information a specifying means for specifying one language from the list of languages, and specifying the language corresponding to the country information from among the specified languages; and setting a voice setting of the smart speaker to the language corresponding to the country information specified by the specifying means. The smart speaker is characterized in that it has a first request means for making a request to the server, the server has a second request means for requesting the smart speaker to change the audio settings based on the request by the first request means, and the smart speaker has a change means for changing the audio settings of the smart speaker based on the request by the second request means. Effect of the Invention
[0009] According to the present invention, The voice settings of the smart speaker can be set to be easy for the user to hear. . [Brief description of the drawings]
[0010] [Figure 1] 1 is a diagram illustrating an example of an overall configuration of an information processing system according to an embodiment of the present invention. [Diagram 2] 2 is a block diagram showing an example of a schematic configuration of hardware of the MFP appearing in FIG. 1. [Diagram 3] 2 is a block diagram showing an example of a schematic configuration of hardware of the smart speaker in FIG. 1. [Figure 4] 2 is a block diagram showing an example of a schematic configuration of hardware of a controller unit of the cloud server appearing in FIG. 1. [Diagram 5] 4 is a block diagram showing a functional configuration of a device control program of the MFP. [Figure 6] 4 is a block diagram showing a functional configuration of a voice device control program of the smart speaker. FIG. [Figure 7] 4 is a block diagram showing a functional configuration of a voice data conversion control program of a cloud server. FIG. [Figure 8] 1 is a control sequence executed in the information processing system at the time of user login according to the first embodiment. [Figure 9] 9 is a flowchart of a login execution process executed by the MFP in steps S801 to S803 in FIG. 8. [Figure 10] 10 is a flowchart of a locale information acquisition process executed by the MFP in step S905 of FIG. [Figure 11] 10 is a flowchart of a volume setting information acquisition process executed by the MFP in step S906 of FIG. [Figure 12] 10 is a flowchart of an audio device control setting change process executed by the cloud server in step S804 of FIG. 8. [Figure 13] 10 is a flowchart of a voice control setting change process executed by the smart speaker in step S805 of FIG. 8. [Figure 14] 13 is a control sequence executed in the information processing system at the time of user login according to the second embodiment. [Figure 15] 13 is a control sequence when a job according to a second embodiment is executed in an information processing system. [Figure 16] 15 is a flowchart of a login execution process executed by the MFP in steps S1401 to S1403 in FIG. [Figure 17] 15 is a flowchart of an available job type changing process executed by the cloud server in step S1404 of FIG. [Figure 18] 16 is a flowchart of a job input process, which is a part of the job execution process executed by the cloud server in step S1506 of FIG. 15. [Figure 19] 13A and 13B are diagrams illustrating switching of the display of the home screen on the operations panel in step S910, which is executed when user authentication is successful in the MFP. [Figure 20] FIG. 13 is a diagram showing a login user information table held in the MFP. [Figure 21] 11A and 11B are diagrams showing a country setting table, a locale code table, a locale code lookup table, a volume setting table, and a volume setting lookup table stored in the MFP. [Figure 22] 11 is a diagram showing an example of voice control information transmitted from the MFP to the cloud server. [Diagram 23] 13 is a diagram showing an example of voice device control information transmitted from a cloud server to a smart speaker. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[0012] <Overall composition> FIG. 1 is a diagram showing an example of the overall configuration of an information processing system 100 according to an embodiment of the present invention.
[0013] 1, an information processing system 100 includes a multifunction peripheral (hereinafter referred to as MFP) 101 having print and scan functions as an information processing device, a smart speaker 102 as a voice control device, and a cloud server 103. The MFP 101 and the smart speaker 102 can communicate with each other via a network 104. In addition, the cloud server 103 can communicate with the MFP 101 and the smart speaker 102 via a gateway 105 and the network 104.
[0014] In the information processing system 100, the cloud server 103 holds services for operating the MFP 101 from the smart speaker 102, such as “job execution” and “job setting.” The cloud server 103 also receives predetermined voice data from the smart speaker 102 via the network 104.
[0015] When the cloud server 103 receives the voice data, it executes a service corresponding to the voice data, and transmits device operation data generated by the service to the MFP 101 via the network 104 .
[0016] When the MFP 101 receives the device operation data from the cloud server 103, it executes processing corresponding to the device operation data. The MFP 101 transmits a response to the executed processing to the cloud server 103 via the network 104.
[0017] When the cloud server 103 receives a response to the executed process from the MFP 101 , it generates response message data and transmits it to the smart speaker 102 via the network 104 .
[0018] When the smart speaker 102 receives the response message data from the cloud server 103, it produces a voice corresponding to the response message data.
[0019] The MFP 101 holds information such as other MFP device data 107 and its own MFP device data 106 .
[0020] The other MFP device data 107 includes account information of the user who uses the service, response notification URL information for notifying the cloud server 103 of the results of each service, and the like.
[0021] The MFP own data 106 includes an IP address, a MAC address, and the like for accessing the MFP 101 via a network.
[0022] The smart speaker 102 holds information such as other smart speaker device data 109 and smart speaker own device data 108.
[0023] The smart speaker and other device data 109 includes account information for using a service, a service activation word, a service URL of the cloud server 103 corresponding to the service activation word, and the like.
[0024] The smart speaker's own data 108 includes an IP address, a MAC address, and the like for accessing the smart speaker 102 via the network.
[0025] The cloud server 103 holds service URL information for the MFP 101 and the smart speaker 102 to use the services of the cloud server 103 via the network.
[0026] Cloud server 103 holds information such as cloud server own device data 110 and cloud server other device data 111 .
[0027] Cloud server own device data 110 includes response notification URL information for MFP 101 to notify cloud server 103 of the results of each service.
[0028] The cloud server and other device data 111 includes the accounts of users who use the service, the IP addresses and MAC addresses of the MFP 101 and smart speaker 102 used in the information processing system 100, and the like.
[0029] The cloud server 103 communicates with the MFP 101 and the smart speaker 102 using the IP addresses and MAC addresses stored in the cloud server other device data 111 .
[0030] Network 104 transmits the voice data acquired by smart speaker 102 to cloud server 103. Further, network 104 transmits various data from cloud server 103 to smart speaker 102, and transmits and receives various data such as print jobs and scan jobs from cloud server 103.
[0031] Gateway 105 is, for example, a wireless LAN router compliant with the IEEE802.11 standard series. However, it may have the ability to operate according to other wireless communication methods. Further, instead of a wireless LAN router, it may be a wired LAN router compliant with the Ethernet standard represented by 10BASE-T, 100BASE-T, 1200BASE-T, etc., and may have the ability to operate according to other wired communication methods. Note that the IEEE802.11 standard series includes a series of standards belonging to IEEE802.11, such as IEEE802.11a and IEEE802.11b.
[0032] Note that the following embodiments do not limit the invention according to the claims, and not all combinations of the features described in the embodiments are essential for the solution means of the invention.
[0033] <Configuration of MFP> FIG. 2 is a block diagram showing a schematic configuration example of the hardware of MFP 101.
[0034] In FIG. 2, MFP 101 includes a controller unit 200, an operation panel 209 connected thereto, a print engine 211, a scanner 213, and a card reader 214.
[0035] Controller unit 200 includes a CPU 202, a RAM 203, a ROM 204, a storage 205, a network I / F 206, a display controller 207, an operation I / F 208, a print controller 210, and a scan controller 212. Each component is connected to a system bus 201 and can communicate with each other.
[0036] The CPU 202 controls the overall operation of the MFP 101. The CPU 202 reads out a control program stored in the ROM 204 or the storage 205, and performs various types of control such as reading control and printing control.
[0037] The RAM 203 is a volatile memory, and is used as a main storage memory and work area for the CPU 202, and as a temporary storage area for loading various control programs stored in the ROM 204.
[0038] The ROM 204 is a non-volatile memory and stores control programs that can be executed by the CPU 202 .
[0039] The storage 205 is a storage device (e.g., a hard disk drive: HDD) with a larger capacity than the RAM 203, and stores print data, image data, various programs, and various setting information (MFP other device data 107 and MFP own device data 106, etc.). The storage 205 may be a solid state drive (SSD) or the like, or may be replaced with another storage device having the same function as a hard disk drive.
[0040] In the MFP 101 of this embodiment, one CPU 202 executes each process shown in the flowcharts described later using one memory (RAM 203), but other configurations are also possible. For example, each process shown in the flowcharts described later can be executed by using multiple CPUs, RAMs, ROMs, and storages in cooperation with each other. Also, some of the processes can be executed using hardware circuits such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).
[0041] Network I / F 206 is an interface for allowing MFP 101 to communicate with external devices via network 104. Furthermore, MFP 101 analyzes print data received via network I / F 206 using a software module (PDL analysis unit, not shown) for analyzing print data stored in storage 205 or ROM 204. The PDL analysis unit generates data to be printed by print engine 211 based on print data expressed in various types of page description languages.
[0042] The display controller 207 is connected to an operation panel 209 and a card reader 214. A screen controlled by an operation I / F 208 is displayed on the operation panel 209. When a user operates the operation panel 209, the CPU 202 obtains an event corresponding to the user operation via the display controller 207.
[0043] The operation panel 209 is configured with a touch panel, and displays a screen controlled by the operation I / F 208, and detects user input to that screen.
[0044] The print controller 210 is connected to the print engine 211. Image data to be printed is transferred to the print engine 211 via the print controller 210. The print engine 211 receives a control command and the image data to be printed, and forms an image based on the image data on a sheet. The printing method of the print engine 211 may be an electrophotographic method or an inkjet method. In the case of the electrophotographic method, an electrostatic latent image is formed on a photoconductor, and then developed with toner, and the toner image is transferred to a sheet, and the transferred toner image is fixed to form an image. On the other hand, in the case of the inkjet method, ink is ejected to form an image on a sheet.
[0045] The scan controller 212 is connected to a scanner 213. The scanner 213 reads an image on a sheet and generates image data. The image data generated by the scanner 213 is stored in the storage 205. Furthermore, in the MFP 101, an image can be formed on a sheet by transferring the image data generated by the scanner 213 to the print engine 211. The scanner 213 has a document feeder (not shown) and can read sheets placed on the document feeder while transporting them one by one.
[0046] When a user brings a smart card close to the card reader 214, the card reader 214 reads information from the smart card and performs user authentication. If the user authentication by the card reader 214 is successful, the user becomes able to use multiple functions of the MFP 101, such as copying, printing, and scanning.
[0047] <Smart speaker configuration> FIG. 3 is a block diagram showing an example of the hardware configuration of the smart speaker 102.
[0048] In FIG. 3, the smart speaker 102 includes a controller unit 300 and associated devices such as a microphone 308 as an audio input device, a speaker 310 as an audio output device, and an LED 312 as a notification device.
[0049] The controller unit 300 includes a CPU 302, a RAM 303, a ROM 304, a storage 305, a network I / F 306, a microphone I / F 307, an audio controller 309, and a display controller 311. Each component is connected to a system bus 301 and can communicate with each other.
[0050] The CPU 302 is a central processing unit that controls the overall operation of the controller unit 300 .
[0051] The RAM 303 is a volatile memory, and is used as a main storage memory and work area for the CPU 302, and as a temporary storage area for expanding various control programs stored in the storage 305.
[0052] The ROM 304 is a non-volatile memory, and stores a startup program for the CPU 302 .
[0053] The storage 305 is a storage device (e.g., an SD card) with a larger capacity than the RAM 303. The storage 305 stores a control program for the smart speaker 102 executed by the controller unit 300. The storage 305 may be replaced with a flash ROM other than the SD card, or may be replaced with another storage device having the same function as the SD card.
[0054] At startup, such as when the power is turned on, the CPU 302 executes a startup program stored in the ROM 304. By executing this startup program, the CPU 302 reads out a control program stored in the storage 305 and loads it on the RAM 303. After the CPU 302 executes the startup program, it subsequently executes the control program loaded on the RAM 303 to perform control. The CPU 302 also stores data used when executing the control program in the RAM 303 and reads and writes it. Various settings and the like required when executing the control program can also be stored in the storage 305, and are read and written by the CPU 302.
[0055] The network I / F 306 is an interface for allowing the smart speaker 102 to communicate with an external device via the network 104. The network I / F 306 includes a circuit and an antenna for performing communication according to a wireless communication method conforming to the IEEE802.11 standard series. However, the network I / F 306 is not limited to a wireless communication method and may be a wired communication method conforming to the Ethernet standard instead of a wireless communication method.
[0056] The microphone I / F 307 is connected to a microphone 308 , converts the voice uttered by the user and input from the microphone 308 into encoded voice data, and stores it in the RAM 303 in response to an instruction from the CPU 302 .
[0057] In this embodiment, the microphone 308 is a small MEMS microphone mounted on a smartphone or the like, but may be replaced with other devices that can capture the user's voice. It is preferable to use three or more microphones 308 arranged at predetermined positions so that the direction of arrival of the voice emitted by the user can be calculated. However, the information processing system 100 according to this embodiment can be realized even with one microphone 308, and the number of microphones is not limited to three or more.
[0058] The audio controller 309 is connected to a speaker 310 , converts audio data into an analog audio signal in response to an instruction from the CPU 302 , and outputs the audio through the speaker 310 .
[0059] The speaker 310 plays a response sound indicating that the smart speaker 102 is responding to a voice uttered by the user, and a voice synthesized by the cloud server 103. The speaker 310 is a general-purpose device for playing voice.
[0060] The display controller 311 is connected to the LED 312, and controls the display of the LED 312 in response to an instruction from the CPU 302. Here, the display controller 311 mainly controls the lighting of the LED 312 to indicate that the smart speaker 102 is correctly inputting the user's voice. The LED 312 is, for example, a blue LED that is visible to the user. The LED 312 is a general-purpose device. Note that in the smart speaker 102, a display device capable of displaying characters and pictures may be used instead of the LED 312.
[0061] <Cloud server hardware configuration> FIG. 4 is a block diagram showing an example of a schematic configuration of hardware of the controller unit 400 of the cloud server 103. As shown in FIG.
[0062] 4, the controller unit 400 includes a CPU 402, a RAM 403, a ROM 404, a storage 405, and a network I / F 406. Each component is connected to a system bus 401 and can communicate with each other.
[0063] The CPU 402 is a central processing unit that controls the overall operation of the controller unit 400 .
[0064] The RAM 403 is a volatile memory.
[0065] The ROM 404 is a non-volatile memory, and stores a startup program for the CPU 402 .
[0066] The storage 405 is a storage device (for example, a hard disk drive: HDD) having a larger capacity than the RAM 403. The storage 405 stores a control program for the cloud server 103, which is executed by the controller unit 400. The storage 405 may be a solid state drive (SSD) or the like, or may be replaced with another storage device having the same function as a hard disk drive.
[0067] At startup, such as when the power is turned on, the CPU 402 executes a startup program stored in the ROM 404. By executing this startup program, the CPU 402 reads out a control program stored in the storage 405 and loads it on the RAM 403. After executing the startup program, the CPU 402 subsequently executes the control program loaded on the RAM 403 to perform control. The CPU 402 also stores data used when executing the control program in the RAM 403 and reads and writes it. Various settings required when executing the control program can also be stored in the storage 405, and are read and written by the CPU 402.
[0068] The network I / F 406 is an interface for communicating with an external device, the cloud server 103, via the network 104.
[0069] <Functional Configuration of the MFP's Device Control Program> FIG. 5 is a block diagram showing the functional configuration of the device control program 500 of the MFP 101.
[0070] The device control program 500 is one of the control programs stored in the aforementioned ROM 204, and is expanded and executed by the CPU 202 on the RAM 203 at the time of its startup.
[0071] As shown in FIG. 5, the device control program 500 includes a data transmission / reception unit 501, a data analysis unit 502, a job control unit 503, a data management unit 504, a display unit 505, an operation target determination unit 506, a scan unit 507, and a printer unit 508.
[0072] The data transmission / reception unit 501 performs data transmission and reception with the smart speaker 102 and the cloud server 103 that are directly or indirectly connected to the network 104 via the network I / F 206 by TCP / IP. The data transmission / reception unit 501 receives the device operation data generated by the cloud server 103. Further, the data transmission / reception unit 501 transmits to the cloud server 103 a job execution result, a screen update notification indicating that the screen display content of the response of the device operation result has been updated, and a job execution state notification indicating the state of the job. The content of the screen update notification and the job execution state notification will be described later using the sequences of FIGS. 8 and 14.
[0073] The data analysis unit 502 converts the device operation data received by the data transmission / reception unit 501, the job parameters and the job start instruction received from the operation target determination unit 506 into commands for each module in the device control program 500 to communicate. Thereafter, the data analysis unit 502 transmits the converted commands to at least one corresponding device among the job control unit 503, the data management unit 504, and the display unit 505.
[0074] The job control unit 503 instructs the printer unit 508 to control the print engine 211 via the print controller 210, and instructs the scan unit 507 to control the scanner 213 via the scan controller 212. For example, when the display unit 505 displays a copy function screen on the operation panel 209 and the operation object determination unit 506 detects pressing of the start key on the screen, the data analysis unit 502 receives job parameters of a copy job and a job start instruction from the operation object determination unit 506. The data analysis unit 502 generates a scan command including scan job parameters and a print command including printer job parameters from the received job parameters. Thereafter, the data analysis unit 502 transmits the generated scan command to the scan unit 507 and transmits the print command to the printer unit 508. As a result, the image data read by the scanner 213 is controlled so that the print engine 211 prints the image data on a sheet. Note that the mechanism of scan and print control is not the main point, so further explanation will be omitted.
[0075] The data management unit 504 stores and manages various data, such as work data generated in the execution of the device control program 500 and setting parameters (MFP setting information) required for controlling each device, in a predetermined area on the RAM 203 and the storage 205. For example, the data management unit 504 stores and manages job data consisting of combinations of parameter setting items and their setting values for various jobs generated by the job control unit 503 described later, and display language setting information, which is information on the language displayed on the operation panel 209. The data management unit 504 also stores and manages authentication information required for communication with the gateway 105 and device information required for communication with the cloud server 103. The data management unit 504 also stores and manages image data of a target to be image-formed by the MFP 101. The data management unit 504 also manages screen control information used by the display unit 505 for screen display control and operation target determination information used by the operation target determination unit 506 for determining an operation target for each screen of the operation panel 209 that the display unit 505 controls to display.
[0076] Further, the data management unit 504 holds and manages a login user information table 2001 and a user login flag. Here, the login user information table 2001 is composed of a list of record information for each user, including a user's identification ID (identification information), a user name, language information used by the user, and information on restrictions on functions available to the user, as shown in FIG. 20. The login user information table 2001 may include smart speaker information for each user, such as the speaking speed (fast to slow), gender (male, female, asexual), voice pitch (high to low), and character (child, elderly, robot). The data management unit 504 also holds and manages a country setting table 2101, a locale code table 2102, and a locale code lookup table 2103 in FIG. 21. The country setting table 2101 is stored in the RAM 203 with one country selected therein.
[0077] Furthermore, the data management unit 504 stores and manages a volume setting table 2104 and a volume setting lookup table 2105 in the RAM 203 .
[0078] Display unit 505 controls operation panel 209 via display controller 207. More specifically, display unit 505 displays UI components (such as buttons, pull-down lists, and check boxes) that can be operated by the user on operation panel 209. Display unit 505 updates the screen of operation panel 209 based on screen display control information and language settings stored in data management unit 504. More specifically, when updating the screen, a language dictionary corresponding to the language setting stored in data management unit 504 is obtained from storage 205, and the screen of operation panel 209 is updated to a screen consisting of text data based on the language dictionary.
[0079] Operation target determination unit 506 acquires the coordinates of a touch on operation panel 209 via operation I / F 208, and determines the UI part located at those coordinates as the target currently being operated by the user. In addition, when a smart card held by the user enters a communication area of card reader 214, the operation target determination unit 506 determines the smart card as the target currently being operated by the user.
[0080] Furthermore, operation target determination unit 506 reads screen display control information corresponding to the UI widget determined as the operation target, and determines the processing content at the time of accepting the operation based on the information. For example, operation target determination unit 506 issues an instruction to display unit 505 to update the display content of the screen of operation panel 209, or transmits job parameters set by a user operation and an instruction to start the job to data analysis unit 502.
[0081] The scan unit 507 executes scanning with the scanner 213 via the scan controller 212 based on the scan job parameter settings sent from the job control unit 503 , and stores the read image data in the data management unit 504 .
[0082] The printer unit 508 executes printing using a print engine 211 via a print controller 210 based on the print job parameter settings sent from the job control unit 503 .
[0083] <Functional configuration of the smart speaker voice device control program> FIG. 6 is a block diagram showing the functional configuration of the audio device control program 600 of the smart speaker 102. As shown in FIG.
[0084] The audio device control program 600 is one of the control programs stored in the storage 305 described above, and is deployed on the RAM 303 and executed by the CPU 302 when the smart speaker 102 is started up.
[0085] As shown in FIG. 6, the voice device control program 600 includes a data transmission / reception unit 601, a data management unit 602, a voice control unit 603, a voice acquisition unit 604, a voice playback unit 605, a display unit 606, a voice operation start detection unit 607, and an utterance end determination unit 608.
[0086] A data transmission / reception unit 601 transmits and receives data by TCP / IP with the MFP 101 and cloud server 103 that are directly or indirectly connected to the network 104, via the network I / F 306. The data transmission / reception unit 601 transmits voice data uttered by the user acquired by a voice acquisition unit 604 to the cloud server 103, and also receives from the cloud server 103 voice synthesis data generated on the cloud server 103 as a response to the user.
[0087] The data management unit 602 stores and manages various data, such as operation data generated during execution of the audio device control program 600, in a predetermined area on the storage 305. The data stored and managed by the data management unit 602 includes volume setting data for audio played by the audio playback unit 605 and authentication information required for communication with the gateway 105. Furthermore, the data management unit 602 stores and manages each device information required for communication with the MFP 101 and the cloud server 103.
[0088] The voice acquisition unit 604 converts the analog voice of the user in the vicinity of the smart speaker 102 acquired by the microphone 308 into voice data, and then temporarily stores the voice data on the RAM 303. The voice acquisition unit 604 converts the analog voice of the user into a predetermined format, such as MP3, and temporarily stores the encoded voice data to be transmitted to the cloud server 103 on the RAM 303. The timing of the start and end of the processing of the voice acquisition unit 604 is managed by the voice control unit 603. Furthermore, the voice data may be encoded in a general-purpose streaming format. In this case, the voice acquisition unit 604 may sequentially transmit the encoded voice data to the data transmission / reception unit 601, and transmit the voice data to at least one of the MFP 101 and the cloud server 103.
[0089] The voice reproduction unit 605 reproduces the synthesized voice data (voice message) received by the data transmission / reception unit 601 on the speaker 310 via the audio controller 309. The timing of the voice reproduction by the voice reproduction unit 605 is managed by the voice control unit 603.
[0090] The display unit 606 performs display on the LED 312 via the display controller 311. For example, when a voice operation start detection unit 607 detects that a voice operation has started, the display unit 606 controls the lighting of the LED 312. The display timing of the display unit 606 is managed by the voice control unit 603.
[0091] The voice operation start detection unit 607 detects a wake word uttered by the user or pressing of an operation start key (not shown) of the smart speaker 102, and transmits an operation start notification to the voice control unit 603. Here, the wake word is a predetermined voice word. The voice operation start detection unit 607 constantly detects the wake word from the analog voice of a user in the vicinity of the smart speaker 102 acquired by the microphone 308. The user can operate the MFP 101 by speaking the wake word and then speaking what he or she wants to do. The voice processing after the voice operation start detection unit 607 detects the wake word will be described later.
[0092] The speech end determination unit 608 determines the end timing of the processing in the voice acquisition unit 604. For example, after the voice acquisition unit 604 acquires the analog voice of the user, when the time when the user does not speak (hereinafter, referred to as blank time) becomes a predetermined time (for example, 3 seconds), it determines that the user's speech has ended, and transmits a speech end notification to the voice control unit 603. The end timing may be determined based on the blank time, or based on a predetermined phrase uttered by the user. For example, "Yes", "No", "OK", "Cancel", "End", "Start", "Start", etc. can be set in advance as the predetermined phrase. Thereby, when any of the predetermined phrases is uttered by the user, it may be determined that the user's speech has ended before the blank time has elapsed. In addition, the end timing may be determined by the cloud server 103 instead of the smart speaker 102, and the end of the speech may be determined based on the meaning and context of the content of the user's speech.
[0093] The voice control unit 603 is the center of control in the voice device control program 600, and controls the other modules in the voice device control program 600 to operate in cooperation with each other. Specifically, it controls the start and end of processing of the voice acquisition unit 604, the voice playback unit 605, and the display unit 606. In addition, after the voice acquisition unit 604 acquires voice data, it controls the data transmission / reception unit 601 to transmit the voice data to the cloud server 103. In addition, after the data transmission / reception unit 601 receives voice synthesis data from the cloud server 103, it controls the voice playback unit 605 to play the voice synthesis data.
[0094] Here, the timing of starting and ending the processing of the voice acquisition unit 604, the voice reproduction unit 605, and the display unit 606 will be described.
[0095] When the voice control unit 603 receives an operation start notification from the voice operation start detection unit 607, it starts the processing of the voice acquisition unit 604. When the voice control unit 603 receives an utterance end notification from the utterance end determination unit 608, it ends the processing of the voice acquisition unit 604.
[0096] For example, suppose that the user utters the wake word and then utters "I want to copy." At this time, the voice operation start detection unit 607 detects the analog voice of the wake word and transmits an operation start notification to the voice control unit 603. Upon receiving the operation start notification, the voice control unit 603 causes the voice acquisition unit 604 to start processing and starts an interactive session with the user. The voice acquisition unit 604 thus starts processing, acquires the analog voice of "I want to copy" uttered by the user, converts it into voice data, and temporarily stores the converted voice data in the RAM 303.
[0097] Thereafter, when the speech end determination unit 608 determines that the blank time after the voice acquisition unit 604 acquires the analog voice of "I want to copy" has reached a predetermined time, it transmits an utterance end notification to the voice control unit 603. Upon receiving the utterance end notification, the voice control unit 603 causes the voice acquisition unit 604 to end the processing. Hereinafter, the state of the smart speaker 102 from when the voice control unit 603 causes the voice acquisition unit 604 to start processing to when it ends the processing is referred to as the speech processing state. The voice control unit 603 controls the display unit 606 to light up the LED 312 during the speech processing state.
[0098] When the voice control unit 603 receives a speech end notification from the speech end determination unit 608, the voice acquisition unit 604 transmits the voice data temporarily stored on the RAM 202 to the cloud server 103 through the data transmission / reception unit 601. After that, the data transmission / reception unit 601 waits for a response message from the cloud server 103. Here, the response message from the cloud server 103 includes, for example, a header section indicating that it is a response and voice synthesis data. When the voice control unit 603 receives the response message from the cloud server 103 through the data transmission / reception unit 601, the voice control unit 603 reproduces the voice synthesis data (for example, "Copy screen will be displayed") included in the response message through the voice reproduction unit 605. After that, when the voice control unit 603 receives an interaction session end notification from the cloud server 103 at the same time as the response message through the data transmission / reception unit 601, the voice control unit 603 reproduces the voice synthesis data (for example, "Copy has been completed") included in the response message through the voice reproduction unit 605. In response to the received interaction session end notification, the voice control unit 603 also terminates the interaction session with the user. Note that a state in which the voice control unit 603 receives an utterance end notification from the utterance end determination unit 608 until it receives an interaction session end notification from the cloud server 103 is called a response processing state. The voice control unit 603 controls the display unit 606 to blink the LED 312 during the response processing state.
[0099] Even after the response processing state ends, while the dialogue session with the user continues, the user can continue to utter what the user wants to do on the MFP 101 after uttering "I want to copy" without uttering the wake word again. The end of the dialogue session is determined by the cloud server 103, and transmits a dialogue session end notification to the smart speaker 102. When the dialogue session end notification is transmitted from the cloud server 103, the voice control unit 603 controls the data transmission and reception unit 601 to end the dialogue session with the user. Hereinafter, the state of the smart speaker 102 from when the voice control unit 603 causes the data transmission and reception unit 601 to end the dialogue session to when the voice control unit 603 causes the data transmission and reception unit 601 to start the next dialogue session is called a standby state. In addition, after the smart speaker 102 is started, it is always in a standby state until the voice control unit 603 receives an operation start notification from the voice operation start detection unit 607. During the standby state, the voice control unit 603 controls the display unit 606 to turn off the LED 312.
[0100] <Functional configuration of the cloud server's voice data conversion control program> FIG. 7 is a block diagram showing the functional configuration of a voice data conversion control program 700 of the cloud server 103. As shown in FIG.
[0101] The audio data conversion control program 700 is one of the control programs stored in the storage 405 described above, and is deployed on the RAM 403 and executed by the CPU 402 when the cloud server 103 is started up.
[0102] 7, the voice data conversion control program 700 includes a data transmission / reception unit 701, a data management unit 702, a device operation data generation unit 703, and a voice data conversion unit 710. The voice data conversion unit 710 also includes a voice recognition unit 705, a morphological analysis unit 706, a group ID determination unit 707, and a voice synthesis unit 708.
[0103] The data transmission / reception unit 701 transmits and receives data by TCP / IP with the MFP 101 and smart speaker 102 connected to the network 104 via the network I / F 406 and an external gateway 105 of the cloud server 103. The data transmission / reception unit 701 receives user voice data from the smart speaker 102. The data transmission / reception unit 701 also transmits a group ID determined by a group ID determination unit 707 and text data generated by a voice recognition unit 705 through voice recognition processing.
[0104] The data management unit 702 stores and manages various data, such as working data generated in the execution of the voice data conversion control program 700 and voice recognition processing data used by the voice data conversion unit 710, in a predetermined area on the storage 405. The voice recognition processing data stored and managed by the data management unit 702 includes an acoustic model and a language model for converting voice data into text by the voice recognition unit 705 and a dictionary for morphological analysis of text by the morphological analysis unit 706. The voice recognition processing data managed by the data management unit 702 also includes a group ID list for determining a group ID by the group ID determination unit 707 and a voice database for voice synthesis by the voice synthesis unit 708. The data management unit 702 also stores and manages each device information required for communication with the smart speaker 102 and the MFP 101.
[0105] When the voice data of the user is input from the data transmission / reception unit 701, the voice recognition unit 705 performs a voice recognition process for converting the voice data into text. In the voice recognition process, the user's voice data is converted into phonemes using an acoustic model, and the phonemes are further converted into actual text data using a language model. In this embodiment, the user's voice data may be in a plurality of languages. Therefore, the voice recognition unit 705 may use a first voice recognition method for determining the language of the input voice data and converting it into text data according to the determined language. In addition, the voice recognition unit 705 may use a second voice recognition method for converting the input voice data into phonemes using acoustic models of a plurality of languages, and converting it into text data for each of the plurality of languages using a language model corresponding to each acoustic model, and outputting it. In the case of the second voice recognition method, voice data input in a plurality of language formats is converted into text data, so that the voice recognition unit 705 outputs a plurality of voice recognition data consisting of text data and its language format as a result of the voice recognition process. Hereinafter, a case in which the second voice recognition method is used in this embodiment will be described.
[0106] In this embodiment, the voice recognition unit 705 recognizes the input voice data in two language formats, Japanese and English. As a result, the voice recognition unit 705 outputs voice recognition data consisting of text data composed of one or more kana characters and the language setting "Japanese", and voice recognition data consisting of text data composed of one or more alphabet characters and the language setting "English".
[0107] However, as long as the voice data is converted into text, the voice recognition process used by the voice recognition unit 705 is not limited to this. Since the details of the voice recognition process are not the main focus, further explanation will be omitted.
[0108] The morpheme analysis unit 706 performs morpheme analysis on the text data included in the voice recognition data output from the voice recognition unit 705 in accordance with the language setting. Specifically, the morpheme analysis unit 706 derives a morpheme string included in the text data from a dictionary having information such as the grammar and parts of speech of the language setting, and further determines the part of speech of each morpheme. The morpheme analysis unit 706 can be realized by using known morpheme analysis software such as JUMAN, Chado, MeCab, etc.
[0109] Since morphological analysis software is a known technique, a detailed description will not be given here.
[0110] <Example 1> <Login control sequence> FIG. 8 shows a control sequence executed in the information processing system 100 at the time of user login according to this embodiment.
[0111] This process is executed by data exchange between the MFP 101, smart speaker 102, and cloud server 103 that constitute the information processing system 100. More specifically, during an authentication operation in which a user logs in to the MFP 101, the MFP 101 notifies the cloud server 103 and smart speaker 102 of setting information held therein, thereby changing the settings of each.
[0112] 8, the MFP 101, cloud server 103, and smart speaker 102 are in a state in which they can communicate with each other. The MFP 101 has completed initial settings, and is in a state in which a home screen that can call up functions such as copy, scan, and print after power-on startup is displayed in a standard language setting. Here, the standard language setting in this embodiment is Japanese.
[0113] Hereinafter, operations executed by the MFP 101 are actually performed by the CPU 202 executing the device control program 500 described above with reference to Fig. 5. Moreover, operations executed by the smart speaker 102 are actually performed by the CPU 302 executing the audio device control program 600 described above with reference to Fig. 6. Similarly, operations executed by the cloud server 103 are actually performed by the CPU 402 executing the audio data conversion control program 700 described above with reference to Fig. 7.
[0114] First, in step S801, the MFP 101 accepts a login operation from a user (first acquisition means). In this embodiment, the login operation refers to, but is not limited to, a user operation of bringing a smart card close to the card reader 214. For example, the login operation may be a user operation of inputting a user name and password, a personal identification number (PIN), or other information associated with the user into the operation panel 209.
[0115] In step S802, first, the MFP 101 performs login processing in response to the login operation accepted in step S801. As a result, if the user authentication is successful, the MFP 101 then acquires voice control information (second acquisition means). Here, the voice control information includes user information of a successfully authenticated user from information held in the storage 205, and information to be notified to the cloud server 103 and the smart speaker 102 from the MFP setting information held in the MFP 101. Furthermore, if the acquired user information includes the language used by the user, the MFP 101 requests the cloud server 103 to change the voice setting of the smart speaker 102 by changing the display language of the operation panel 209 to the language used by the user (first request means).
[0116] In step S803, the MFP 101 transmits the voice control information acquired in step S802 to the cloud server 103 via the network I / F 407. Details of the processes in steps S801 to S803 will be described later with reference to FIG.
[0117] In step S804, when the cloud server 103 receives the voice control information transmitted in step S803 at the data transmission / reception unit 701, the cloud server 103 changes the setting value related to the voice data processing (hereinafter, referred to as voice device control information) in accordance with the voice control information. Details of the process in step S804 will be described later with reference to the flowchart in FIG.
[0118] In step S805, the cloud server 103 transmits the audio device control information changed in step S804 to the smart speaker 102 via the data transmitter / receiver 701. As a result, the cloud server 103 requests the smart speaker 102 to change the audio settings (request means, second request means).
[0119] In step S806, when the smart speaker 102 receives the audio device control information transmitted in step S805 at the data transmission / reception unit 601, the smart speaker 102 changes the settings of the audio control unit 603, such as the locale setting and the volume setting, in response to the information (changing means). Details of the process in step S805 will be described later with reference to the flowchart in FIG.
[0120] Fig. 9 is a flowchart of login execution processing executed by MFP 101 in steps S801 to S803 in Fig. 8. As described below, the processes of steps S901 and S907 correspond to the processes of steps S801 and S803, respectively, and the processes of the other steps in Fig. 9 correspond to the process of step S802.
[0121] In step S901, the process of step S801 is executed. That is, when the smart card held by the user enters the communication area of the card reader 214, the operation target determination unit 506 detects the smart card by the card reader 214 and acquires the user information. Next, the operation target determination unit 506 checks the acquired user information against the login user information table 2001 held in the data management unit 504.
[0122] In step S902, if there is matching information as a result of the comparison in step S901, the data analysis unit 502 determines that the user authentication was successful, and transitions to step S903. If not, the data analysis unit 502 determines that the user authentication was not successful, and transitions to step S912.
[0123] In step S 903 , the data management unit 504 changes the user login flag to ON, which indicates that the user is in a logged-in state, and stores the changed user login flag in the RAM 203 .
[0124] In step S904, the data management unit 504 obtains user information from the login user information table 2001. For example, if the login user is a user with an identification ID of 1, the data management unit 504 obtains record information with the identification ID of 1 from the login user information table 2001 as user information and stores it in the RAM 203.
[0125] In step S905, the data analysis unit 502 executes a locale information acquisition process to acquire locale information from the user information acquired by the data management unit 504 in step S904 and the MFP setting information held in the MFP 101. Details of the locale information acquisition process will be described later with reference to the flowchart in FIG. 10. The data analysis unit 502 stores the acquired locale information in the RAM 203.
[0126] In step S906, the data analysis unit 502 acquires the volume setting information from the MFP setting information held in the MFP 101. The volume setting information acquisition process will be described in detail later with reference to the flowchart in FIG. 11. The data analysis unit 502 stores the acquired volume setting information in the RAM 203.
[0127] In step S907, the process of step S803 is executed. That is, the data transmission / reception unit 501 transmits the voice control information including the user login flag, user information, locale information, and volume setting information stored in the RAM 203 in steps S903 to S906 to the cloud server 103. In this embodiment, the Json data format shown in FIG. 22 is used for this transmission, but the data format is not limited to this. For example, a data format such as XML may be used. In addition, when smart speaker information is included in the login user information table, the smart speaker information may be included in the voice control information.
[0128] In step S908, the data analysis unit 502 determines whether or not the language used is set in the user information acquired by the data management unit 504 in step S904. If it is set, the process proceeds to step S909. If it is not set, the process proceeds to step S910.
[0129] In step S909, data analysis unit 502 changes display language setting information, which is information about the language to be displayed on operation panel 209 and is stored and managed in data management unit 504, based on the language information set in the user information.
[0130] In step S910, display unit 505 refers to the display language setting information in RAM 203, switches the display of the home screen of operation panel 209 via display controller 207, and ends this process. As a result, the display language of the home screen of operation panel 209 switches to the language used by the logged-in user. This switching will be described later with reference to FIG.
[0131] In step S911, the display unit 505 controls the display controller 207 to display a login error occurrence screen on the operation panel 209, and then ends this process.
[0132] FIG. 10 is a flowchart of the locale information acquisition process executed by the MFP 101 in step S905 of FIG.
[0133] In step S1001, language information is obtained from user information, that is, record information stored in RAM 203 by data management unit 504 in step S904. Here, the following describes a case where a user with an identification ID of 1 is logged in and "En" is obtained in step S1001 as language information associated with the identification ID of 1 from the login user information table in Fig. 20.
[0134] In step S 1002 , the data management unit 504 acquires the country setting information selected in the country setting table 2101 stored and managed in the RAM 203 .
[0135] In step S1003, the data analysis unit 502 judges whether the locale information of the combination of the used language information "En" and the country setting information acquired in steps S1001 and S1002 is in the locale code table 2102 stored in the RAM 203. If the result of this judgment is that the locale information is in the combination, the process proceeds to step S1005, and if the result is not in the combination, the process proceeds to step S1004. For example, if the country setting information acquired in step S1002 is "America", it is judged that "en-US: English (America)", which is a combination of the country setting information acquired in step S1002 and the used language information "En", is in the locale code table 2102. Also, if the country setting information acquired in step S1002 is "Canada", it is judged that "en-CA: English (Canada)", which is a combination of the country setting information acquired in step S1002 and the used language information "En", is in the locale code table 2102. On the other hand, if the country setting information acquired in step S1002 is "Japan", it is judged that the locale information of the combination of the country setting information acquired in step S1002 and the used language information "En" is not in the locale code table 2102.
[0136] In step S1004, the data analysis unit 502 determines whether or not locale information corresponding to "En", which is the used language information acquired in step S1001, is in the locale code lookup table 2103 stored in the RAM 203. As shown in Fig. 21, the locale code lookup table 2103 contains locale information "en-US" corresponding to "En", so in this case, the process proceeds to step S1005. On the other hand, if the used language information acquired in step S1001 is not in the locale code lookup table 2103, the process proceeds to step S1006.
[0137] In step S1005, the data analysis unit 502 acquires the locale information that was in the locale code table 2102 in step S1003 or the locale information that was in the locale code lookup table 2103 in step S1004. At this time, the locale information that has been held as the locale setting is held in the RAM 203 as the default locale information, and this process ends.
[0138] In step S1006, the data analysis unit 502 acquires the default locale information, and this process ends.
[0139] FIG. 11 is a flowchart of the volume setting information acquisition process executed by the MFP 101 in step S906 of FIG.
[0140] In step S 1101 , the data analysis unit 502 acquires information of the volume setting table 2104 held in the RAM 203 .
[0141] In step S1102, the data analysis unit 502 initializes a counter n in the RAM 203 to 0.
[0142] In step S 1103 , the data analyzer 502 obtains the record data in the nth row of the volume setting table 2104 .
[0143] In step S 1104 , the data analysis unit 502 adds the volume setting value in the record data of the nth row of the volume setting table 2104 to the volume total value prepared in the RAM 203 .
[0144] In step S1105, the data analysis unit 502 increments the counter n in the RAM 203 by one.
[0145] In step S1106, if n is N-1, the data analysis unit 502 determines that the reference to the record data of all rows in the volume setting table 2104 has been completed, and proceeds to step S1107. On the other hand, if n is less than N-1, the process returns to step S1103 to continue the reference to the record data. Note that N is the total number of record data in the volume setting table 2104 (7 in this embodiment).
[0146] In this manner, the operations from steps S1103 to S1106 are repeated to calculate a total volume value that is the sum of the volume setting values in all record data.
[0147] In step S1107, the data analysis unit 502 divides the calculated volume sum by N to calculate an average volume value.
[0148] In step S1108, the data analysis unit 502 acquires, from the volume setting lookup table 2105, an output value when the volume average value calculated in step S1107 is used as an input value.
[0149] In step S1109, the data analysis unit 502 acquires the output value acquired in step S1108 as volume setting information, and this process ends.
[0150] According to the process in Fig. 11, the average value of the notification sounds and warning sounds stored on the MFP 101 side in the volume setting table 2104 in Fig. 21 is calculated, and the volume required on the smart speaker 102 side is estimated and set from the average value. This makes it possible to appropriately set the volume on the smart speaker 102 side.
[0151] FIG. 12 is a flowchart of the audio device control setting change process executed by the cloud server 103 in step S804 of FIG.
[0152] This process is started by the CPU 402 when the data transmitter / receiver 701 receives, via the network I / F 406, the voice control information transmitted by the data transmitter / receiver 501 in step S907.
[0153] In step S1201, the voice recognition unit 705 checks whether the user login flag of the voice control information received by the data transmission / reception unit 701 is ON or OFF. If the check result shows that the flag is ON (YES in step S1201), the process proceeds to step S1202, whereas if the flag is OFF (NO in step S1201), the process proceeds to step S1209.
[0154] In step S1202, the voice recognition unit 705 determines whether or not locale information is included in the voice control information received by the data transmission / reception unit 701. If the result of the determination indicates that locale information is included (YES in step S1202), the process proceeds to step S1203, whereas if the result indicates that locale information is not included (NO in step S1202), the process proceeds to step S1205.
[0155] In step S1203, based on the locale information included in the voice control information, the voice recognition unit 705 changes the locale setting stored in the RAM 403. At this time, the locale setting before the change is stored in the RAM 403 as the standard language setting.
[0156] In step S1204, the data management unit 702 stores the locale setting changed in step S1203 in the RAM 403 as data to be transmitted to the smart speaker 102.
[0157] In step S1205, the voice recognition unit 705 judges whether or not the voice control information received by the data transmission / reception unit 701 includes volume setting information. If the judgment result indicates that the volume setting information exists (YES in step S1205), the process proceeds to step S1206. If the volume setting information does not exist (NO in step S1205), the process proceeds to step S1207.
[0158] In step S1206, the data management unit 702 changes the volume setting stored in the RAM 403 as data to be transmitted to the smart speaker 102 to the volume setting information corresponding to the voice control information. At this time, the volume setting before the change is stored in the RAM 403 as the standard volume setting.
[0159] In step S1207, the voice recognition unit 705 judges whether or not there is available function restriction information in the user information of the voice control information received by the data transmission / reception unit 701. If there is available function restriction information (YES in step S1207), the process proceeds to step S1208. If there is no available function restriction information (NO in step S1207), the process proceeds to step S1210.
[0160] In step S1208, the data management unit 702 changes the job type restriction settings stored in the RAM 403 based on the available function restriction information included in the voice control information. At this time, the job type restriction settings before the change are stored as standard job type settings in the RAM 403. Then, the process proceeds to step S1210.
[0161] In step S1209, the standard language setting and the standard volume setting are acquired from the RAM 403. After that, based on the acquired standard language setting and the standard volume setting, the locale setting and the volume setting stored in the RAM 403 as data to be transmitted to the smart speaker 102 are changed, and the process proceeds to step S1210. In addition, the job type restriction setting stored in the RAM 403 is changed to the standard job type setting stored in the RAM 403.
[0162] In step S1210, the voice recognition unit 705 determines whether data to be transmitted to the smart speaker 102 exists in the RAM 403. If the data exists (YES in step S1210), the process proceeds to step S1211. If the data does not exist (NO in step S1210), the process ends.
[0163] In step S1211, the data transmitter / receiver 701 transmits the data held in the RAM 403 as data to be transmitted to the smart speaker 102 from the network I / F 406 as audio device control information, and ends this process. The data format for transmission at this time is the JSON data format shown in Fig. 23 as an example, but is not limited to this. For example, a format such as XML may also be used.
[0164] FIG. 13 is a flowchart of the voice control setting change process executed by the smart speaker 102 in step S805 of FIG.
[0165] This process is started by the CPU 302 when the data transmitter / receiver 601 receives, via the network I / F 306, the audio device control information transmitted by the data transmitter / receiver 701 in step S1211.
[0166] In step S1301, the audio control unit 603 determines whether a locale setting is present in the audio device control information received by the data transmission / reception unit 601. If the determination result indicates that a locale setting is present (YES in step S1301), the process proceeds to step S1302, whereas if the determination result indicates that a locale setting is not present (NO in step S1301), the process proceeds to step S1303.
[0167] In step S1302, the audio control unit 603 changes the locale setting stored in the RAM 303 to the locale setting of the audio device control information.
[0168] In step S1303, the audio control unit 603 determines whether or not a volume setting is present in the audio device control information received by the data transmission / reception unit 601. If the determination result indicates that there is a volume setting (YES in step S1303), the process proceeds to step S1304, whereas if there is no volume setting (NO in step S1301), the process ends.
[0169] In step S1304, the audio control unit 603 changes the volume setting stored in the RAM 303 to the volume setting in the audio device control information, and ends this process.
[0170] FIG. 19 shows the display switching of the home screen on the operation panel 209 in step S910, which is executed when the user authentication in the MFP 101 is successful.
[0171] Here, a case will be described in which, with a home screen 1901 in Japanese, which is the standard language setting, displayed on operation panel 209, the user brings a smart card close to card reader 214, and as a result, MFP 101 successfully authenticates the user.
[0172] If the used language information included in the user information acquired from the smart card is "En", in step S909, the display language setting information in MFP 101 is changed to English. In this case, in step S910, display unit 505 controls display controller 207 to switch home screen 1901 displayed on operation panel 209 to home screen 1902 whose display language is English.
[0173] On the other hand, if the used language information included in the user information acquired from the smart card is "Jp", in step S909, the display language setting information in MFP 101 is changed to Japanese. In this case, in step S910, display unit 505 controls display controller 207 to keep home screen 1901 displayed on operation panel 209 as home screen 1901 with the display language set to Japanese.
[0174] As described above, in this embodiment, it is possible to determine the wake word in the smart speaker 102 and perform voice recognition in the cloud server 103 in the language used by the logged-in user, thereby improving usability.
[0175] <Example 2> In the first embodiment, user authentication at the time of logging in to the MFP 101 is executed within the MFP 101. In contrast, in the present embodiment, user authentication at the time of logging in to the MFP 101 is executed by the cloud server 103. As a result, in the present embodiment, it becomes possible to execute on the cloud server 103 the determination of the job types available for each logged-in user, which was conventionally determined by the MFP 101.
[0176] <Login control sequence> FIG. 14 shows a control sequence executed in the information processing system 100 at the time of user login according to this embodiment.
[0177] This process is executed by data exchange between the MFP 101, smart speaker 102, and cloud server 103 that configure the information processing system 100. More specifically, during an authentication operation in which a user logs in to the MFP 101, the MFP 101 notifies the cloud server 103 of setting information held therein, whereby the settings are changed.
[0178] 14, the MFP 101 and the cloud server 103 are in a state in which they can communicate with each other. The MFP 101 has completed initial settings, and is in a state in which the home screen from which functions such as copy, scan, and print can be called up is displayed in the standard language setting after the power is turned on and startup is started. Here, the standard language setting in this embodiment is Japanese.
[0179] As in the first embodiment, hereinafter, operations executed by the MFP 101 are actually performed by the CPU 202 executing the device control program 500 described above with reference to Fig. 5. Moreover, operations executed by the smart speaker 102 are actually performed by the CPU 302 executing the audio device control program 600 described above with reference to Fig. 6. Similarly, operations executed by the cloud server 103 are actually performed by the CPU 402 executing the audio data conversion control program 700 described above with reference to Fig. 7.
[0180] First, in step S1401, the MFP 101 accepts a login operation from a user. In this embodiment, the login operation refers to, but is not limited to, a user operation of bringing a smart card close to the card reader 214. For example, the login operation may be a user operation of inputting a user name and password, a personal identification number (PIN), or other information associated with the user into the operation panel 209.
[0181] In step S1402, first, the MFP 101 performs login processing in response to the login operation accepted in step S1401. As a result, if the user authentication is successful, the MFP 101 then acquires voice control information. Here, the voice control information includes user information of a successfully authenticated user from information held in the storage 205, and information to be notified to the cloud server 103 and the smart speaker 102 from the MFP setting information held in the MFP 101. Furthermore, if the acquired user information includes the language used by the user, the MFP 101 changes the display language of the operation panel 209 to the language used by the user.
[0182] In step S1403, the MFP 101 transmits the voice control information acquired in step S1402 to the cloud server 103 via the network I / F 407. Details of the processes in steps S1401 to S1403 will be described later with reference to FIG.
[0183] In step S1404, when the cloud server 103 receives the voice control information transmitted in step S803 at the data transmission / reception unit 701, the cloud server 103 changes the available job type of the user who performs voice operation on the smart speaker 102 in accordance with the voice control information. Details of the process in step S1404 will be described later with reference to the flowchart in FIG.
[0184] Although not shown in FIG. 14, in this embodiment as well, audio device control information is generated in response to audio control information by the same process as in FIG.
[0185] <Job execution control sequence> FIG. 15 shows a control sequence during job execution according to this embodiment, which is executed in the information processing system 100.
[0186] This process is executed by data exchange between the MFP 101, smart speaker 102, and cloud server 103 that constitute the information processing system 100. More specifically, in this control sequence, when the smart speaker 102 receives a voice operation from a user, the MFP 101 executes each process in response to the voice operation, and then the smart speaker 102 returns a response indicating the execution result to the user by voice.
[0187] In the example of the sequence shown in Fig. 15, the control sequence at the time of user login in Fig. 14 has ended, and the smart speaker 102, MFP 101, and cloud server 103 are in a state in which they can communicate with each other. Furthermore, the MFP 101 is in a state in which it displays a home screen 1901 in the language used by the logged-in user. Also, the voice operation start detection unit 607 of the smart speaker 102 is in a state in which it constantly detects a wake word from the analog voice of a user in the vicinity of the smart speaker 102 with the microphone 308.
[0188] First, in step S1501, the smart speaker 102 detects a voice operation start instruction operation by the user with the voice operation start detection unit 607. In this embodiment, the voice operation start instruction operation refers to, but is not limited to, an operation in which the user speaks a wake word to the smart speaker 102. For example, the voice operation start instruction operation may be a user pressing an operation start key of the smart speaker 102.
[0189] In step S1502, the smart speaker 102 turns on the LED 312 on the display unit 606 to indicate that the smart speaker 102 is in a speech processing state, and at the same time causes the voice acquisition unit 604 to start processing, thereby starting an interactive session with the user.
[0190] In step S1503, the smart speaker 102 detects a job execution instruction operation by the user with the voice acquisition unit 604 (detection means). Here, the job execution instruction operation refers to an operation in which the user vocalizes, for example, "Make 4 copies" or "Four Copies" during the speech processing state in order to cause the information processing system 100 to execute a job. When the voice acquisition unit 604 (conversion means) acquires the analog voice of the job execution instruction operation, it converts it into voice data and temporarily stores it on the RAM 303. After that, if a predetermined blank time has elapsed and the speech end determination unit 608 determines that the user's speech has ended, the process proceeds to step S1504.
[0191] In step S1504, the smart speaker 102 blinks the LED 312 on the display unit 606 to indicate that the smart speaker 102 is in a response processing state, and at the same time, causes the voice acquisition unit 604 to end the processing.
[0192] In step S1505, the data transmitter / receiver 601 of the smart speaker 102 transmits to the cloud server 103 the voice data temporarily stored on the RAM 303 in step S1503.
[0193] In step S1506 , the cloud server 103 starts job execution processing based on the voice data received by the data transmission / reception unit 701 .
[0194] The job execution process is made up of a voice recognition process, a language determination process, and a job input process which is executed thereafter.
[0195] The voice recognition process is a process in which the voice recognition unit 705 outputs multiple voice recognition data consisting of text data and its language format from the voice data received by the data transmission / reception unit 701 using the second voice recognition method described above.
[0196] The language determination process is a process in which the morphological analysis unit 706 (identification means) morphologically analyzes the text data included in the voice recognition data output from the voice recognition process according to the language setting. This identifies the job type and various keywords of the job instructed to be executed by the user in the job execution instruction operation in step S1503.
[0197] The job submission process performs operation determination processing, job information generation processing, available job determination processing, and transmission of job information to the MFP 101. Thereafter, the job submission process further performs reception of a job execution start notification and a job execution end notification from the MFP 101, voice synthesis processing in response to these notifications, and transmission of the voice synthesis data to the smart speaker 102. Details of the job submission process will be described later using the flowchart in FIG.
[0198] In step S1507, the MFP 101 performs processing according to the job information with language settings transmitted from the cloud server 103 in the job execution processing started in step S1506.
[0199] In step S1508, the data transmission / reception unit 601 of the smart speaker 102 receives the voice synthesis data transmitted from the cloud server 103 in the job execution process started in step S1506.
[0200] In step S1509, the smart speaker 102 causes the voice playback unit 605 to play back through the speaker 310 the synthesized voice data received in step S1508 (for example, the synthesized voice data saying “Start copying”).
[0201] In step S1510, the smart speaker 102 receives the voice synthesis data and the interaction session end notification transmitted from the cloud server 103 in the job execution process started in step S1506, at the data transmission / reception unit 601. Here, the interaction session end notification is a notification from the cloud server 103 to the smart speaker 102 instructing it to end the interaction session with the user.
[0202] In step S1511, the smart speaker 102 causes the voice playback unit 605 to play back through the speaker 310 the synthesized voice data received in step S1510 (for example, synthesized voice data saying “Copying has been completed”).
[0203] In step S1512, the smart speaker 102 turns off the LED 312 to indicate on the display unit 606 that the smart speaker 102 has finished the response processing state and is in a standby state.
[0204] In step S1513, the smart speaker 102 transitions to a standby state and ends the interactive session with the user.
[0205] 15, even if the smart speaker 102 is in a response processing state (i.e., the LED 312 is flashing), the voice operation start detection unit 607 detects the wake word. This allows the user to forcibly end the interaction session by uttering "cancel" or "stop" as a function invocation instruction operation following utterance of the wake word to the smart speaker 102 in the response processing state.
[0206] Fig. 16 is a flowchart of login execution processing executed by MFP 101 in steps S1401 to S1403 in Fig. 14. As described below, the processes of steps S1601 and S1606 correspond to the processes of steps S1401 and S1403, respectively, and the processes of the other steps in Fig. 16 correspond to the process of step S1402.
[0207] In step S1601, the process of step S1401 is executed. That is, when the smart card held by the user enters the communication area of the card reader 214, the operation target determination unit 506 detects the smart card by the card reader 214 and acquires the user information. Next, the operation target determination unit 506 checks the acquired user information against the login user information table 2001 held in the data management unit 504.
[0208] In step S1602, if there is matching information as a result of the comparison in step S1601, the data analysis unit 502 determines that the user authentication was successful, and transitions to step S1603. If not, the data analysis unit 502 determines that the user authentication was not successful, and transitions to step S1607.
[0209] In step S 1603 , the data management unit 504 changes the user login flag to ON, which indicates that the user is in a logged-in state, and stores the changed user login flag in the RAM 203 .
[0210] In step S1604, the data management unit 504 obtains user information from the login user information table 2001. For example, if the login user is a user with an identification ID of 1, the data management unit 504 obtains record information with the identification ID of 1 from the login user information table 2001 as user information and stores it in the RAM 203.
[0211] In step S1605, the data analysis unit 502 acquires the job types available to the logged-in user from the user information acquired by the data management unit 504 in step S1604. For example, if the logged-in user is a user with an identification ID of 1, SEND / PRINT / SCAN / FAX is acquired as the information on the job types available to the logged-in user, and stored in the RAM 203. The data analysis unit 502 also acquires locale information and volume setting information by processing similar to steps S905 and S906, and stores them in the RAM 203.
[0212] In step S1606, the process of step S1403 is executed. That is, data transmission / reception unit 501 transmits to cloud server 103 the audio control information including the user login flag, the information on the job types available to the logged-in user, the locale information, and the volume setting information stored in RAM 203 in steps S1603 and S1605. Then, this process ends.
[0213] In step S1607, the display unit 505 controls the display controller 207 to display a login error occurrence screen on the operation panel 209, and then ends this process.
[0214] FIG. 17 is a flowchart of the available job type changing process executed by the cloud server 103 in step S1404 of FIG.
[0215] This process is started by the CPU 402 when the data transmitter / receiver 701 receives, via the network I / F 406, the voice control information transmitted by the data transmitter / receiver 501 in step S1606.
[0216] In step S1701, the voice recognition unit 705 checks whether the user login flag is ON or OFF in the voice control information received by the data transmission / reception unit 701. If the check result shows that the flag is ON (YES in step S1701), the process proceeds to step S1702, whereas if the flag is OFF (NO in step S1701), the process proceeds to step S1704.
[0217] In step S1702, the voice recognition unit 705 determines whether or not there is information on the job types available to the logged-in user in the voice control information received by the data transmission / reception unit 701. If there is (YES in step S1702), the process proceeds to step S1703, whereas if there is not (NO in step S1702), the process proceeds to step S1705.
[0218] In step S1703, the data management unit 702 changes the settings of the available job types stored in the RAM 403 based on the information on the available job types of the logged-in user contained in the voice control information. At this time, the settings of the available job types before the change are stored in the RAM 403 as standard job type settings, and this process ends.
[0219] In step S1704, the settings of the available job types held in the RAM 403 are changed to the standard job type settings held in the RAM 403, and this process ends.
[0220] FIG. 18 is a flowchart of a job input process, which is part of the job execution process executed by the cloud server 103 in step S1506 of FIG.
[0221] This process is executed after the cloud server 103 receives voice data of a job execution instruction operation from the user from the smart speaker 102 in step S1505, converts this into text data through voice recognition processing, and performs morphological analysis of the text data through language determination processing.
[0222] First, in step S1801, the CPU 402 executes an operation determination process. Specifically, it is determined whether or not the morphological analysis result in the language determination process includes all the job mandatory setting keywords in the group ID list stored in the RAM 403. The job mandatory setting keywords are keywords that indicate settings that must be set by the user when performing the job execution instruction operation in step S1503. For example, if the morphological analysis result includes "EMAILSEND" as the job type, the morphological analysis result must include a keyword (e.g., "CNF00004") that indicates the destination. Note that the job mandatory setting keywords differ depending on the job type, and there may be some job types that do not have a job mandatory setting keyword and some job types that have multiple job mandatory setting keywords.
[0223] If it is determined in step S1801 that all the job essential setting keywords are present, the process proceeds to step S1802, where the CPU 402 (generation means) executes job information generation processing based on the morphological analysis results. Through this processing, job information with language settings is generated, which is device operation data for the MFP 101 to execute the job.
[0224] In step S1803, the CPU 402 executes an available job determination process. Specifically, if the setting of available job types held in the RAM 403 includes the job type included in the morphological analysis result, the process proceeds to step S1804, and if not, the process proceeds to step S1809.
[0225] In step S1804, the data transmission / reception unit 701 (transmission means) transmits the job information with language setting generated in step S1802 to the MFP 101 via the network I / F 406. Upon receiving this job information with language setting, the MFP 101 starts the process of step S1507 in FIG.
[0226] In step S1805, the data transmission / reception unit 701 waits for reception of a job end notification from the MFP 101 via the network I / F 406, and transitions to step S1806 when this is received. Note that, although not shown in Fig. 18, when the data transmission / reception unit 701 receives a job execution start notification from the MFP 101, the CPU 402 may generate a job start voice, which is a text message corresponding to the job start notification, and transmit the job start voice to the smart speaker 102. An example of the job start voice is "The job has started."
[0227] When a job end notification is received in step S1806, CPU 402 generates a job end voice message corresponding to the received job end notification and sets it as voice data in RAM 403. If the job end notification is a notification that the job has been normally ended, a voice message stating "Job has been ended" is generated as the job end voice message. On the other hand, if the job end notification is a notification that a paper jam or other error has occurred in MFP 101, a voice message stating "Ended with an error" is generated as the job end voice message.
[0228] In step S1807, the data transmitter / receiver 701 transmits the voice data set on the RAM 403 to the smart speaker 102 via the network I / F 406, and ends this process. Here, the voice data set on the RAM 403 refers to the voice data set on the RAM 403 in step S1806 or in any of steps S1808 and S1809 described below.
[0229] If it is determined in step S1801 that all the required job setting keywords are not present, the process proceeds to step S1808. In step S1808, the CPU 402 generates a job setting prompting voice message, which is a text message for prompting the input of the required job setting keywords, and sets the message as voice data in the RAM 403. One example of the job setting prompting voice message is "Please enter the destination address" when the user has specified EMAILSEND but has not specified an address.
[0230] If it is determined in step S1803 that the job type is not available to the logged-in user, the process proceeds to step S1809. In step S1809, the CPU 402 generates a job unavailable voice message, which is a text message notifying that the job is unavailable, and sets the message in the voice data in the RAM 403. An example of a job unavailable voice message is "That function is not available" when the user specifies SEND, which is unavailable, but no destination is specified.
[0231] As described above, it becomes possible for the cloud server 103 to determine the types of jobs available to each logged-in user.
[0232] In the information processing system 100 according to the present embodiment, the MFP 101 and the cloud server 103 are separate devices, but they may be an integrated information processing device.
[0233] Furthermore, in the information processing system 100 according to this embodiment, the cloud server 103 manages the settings of the smart speaker 102, but the settings of the smart speaker 102 may be managed by a server other than the cloud.
[0234] (Other embodiments) The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the recording medium on which the program is stored constitute the present invention. [Explanation of symbols]
[0235] 100 Information Processing Systems 101 MFP 102 Smart Speaker 103 Cloud Server 104 Network 105 Gateway 214 Card Reader 500 Device Control Program 600 Voice device control program 700 Voice data conversion control program
Claims
1. An information processing device that cooperates with a smart speaker, A first acquisition means for acquiring identification information of a user; A second acquisition means for acquiring language information and country information associated with the acquired identification information; a language identification means for identifying one language from the acquired language information and identifying the language corresponding to the country information from among the identified languages; a requesting means for requesting the smart speaker to set a voice setting of the smart speaker to the language corresponding to the country information identified by the identifying means; An information processing device comprising:
2. 2. The information processing apparatus according to claim 1, wherein the audio settings include at least a volume setting.
3. The device further includes an operation panel that can be operated by a user, 3. The information processing apparatus according to claim 1, further comprising: a language display unit configured to change a language displayed on the operation panel based on the language information acquired by the second acquisition unit.
4. 4. The information processing apparatus according to claim 1, wherein the first acquisition means acquires the identification information of the user by short-distance wireless communication.
5. An information processing system in which an information processing device, a smart speaker, and a server that manages settings of the smart speaker cooperate with each other, The information processing device includes: A first acquisition means for acquiring identification information of a user; A second acquisition means for acquiring language information and country information associated with the acquired identification information; a language identification means for identifying one language from the acquired language information and identifying the language corresponding to the country information from among the identified languages; a first request means for requesting the server to set a voice setting of the smart speaker to the language corresponding to the country information identified by the identification means; The server, a second request means for requesting the smart speaker to change a sound setting based on a request by the first request means; The smart speaker includes: An information processing system comprising: a change means for changing a sound setting of the information processing system itself based on a request made by the second request means.
6. A method for controlling an information processing device that cooperates with a smart speaker, comprising: A first acquisition step of acquiring identification information of a user; a second acquisition step of acquiring language information and country information associated with the acquired identification information; a step of identifying one language from the acquired language information, and identifying the language corresponding to the country information from among the identified languages; a request step of requesting the smart speaker to set the voice setting of the smart speaker to the language corresponding to the country information identified in the identification step; A control method comprising the steps of:
7. A control method for an information processing system in which an information processing device, a smart speaker, and a server that manages settings of the smart speaker cooperate with each other, comprising: In the information processing device, A first acquisition step of acquiring identification information of a user; a second acquisition step of acquiring language information and country information associated with the acquired identification information; a step of identifying one language from the acquired language information, and identifying the language corresponding to the country information from among the identified languages; A first request step of requesting the server to set the audio setting of the smart speaker to the language corresponding to the country information identified in the identification step; In the server, A second request step is executed to request the smart speaker to change a sound setting based on the request in the first request step; In the smart speaker, a control method comprising: executing a change step of changing a sound setting of the own device based on the request in the second request step;
8. A program for executing the control method according to claim 6 or 7.
Citation Information
Patent Citations
Data display system, terminal equipment and data display method
JP2006065520A
Multi-function peripheral, authentication server and system
JP2011014996A
Portable terminal device, and method and program for displaying job execution setting screen in the same
JP2014215927A
Image formation apparatus
JP2015079432A
Information processing device and program
JP2018047643A
