Image processing device, program, and server

Abstract

This image processing device that can cooperate with a voice device having a voice notification function comprises: an image processing unit that performs image processing; and a display unit that displays information related to the image processing, the display unit displaying, in synchronization with the voice notification, character information corresponding to voice notification related to the image processing from the voice device and / or an information code that can be converted into character information corresponding to the voice notification.

Description

Image processing device, program and server

[0001] The present disclosure relates to an image processing device, a program, and a server.

[0002] In recent years, voice assistant systems that receive operational instructions given by a user through voice and cause various devices to operate in accordance with the operational instructions have become increasingly popular. In such voice assistant systems, a voice including an operational instruction given by the user is picked up by an audio device such as a smart speaker, converted into audio information, and transmitted to a server. The server then analyzes the audio information, extracts the operational instruction contained in the audio information, and transmits the operational instruction information corresponding to the extracted operational instruction to a target device, thereby causing the device to operate.

[0003] Such control of operation by a voice assistant system is also applied to image processing devices such as printers. For example, Patent Document 1 describes an invention in which a user can use a printing service by issuing a print instruction to a printer via a smart speaker. In this invention, if there are options related to the user's print instruction, such as multiple templates that can be used when printing text, the smart speaker notifies the user of the options by voice, and the user speaks the selected option, causing a server to receive information about the option via the smart speaker, and the server then transmits operation instruction information corresponding to the option to the printer.

[0004] Japanese Patent Application Laid-Open No. 2021-163163

[0005] The image processing device of the first aspect is an image processing device that can be linked with an audio device having an audio notification function, and is equipped with an image processing unit that performs image processing, and a display unit that displays information related to the image processing, and the display unit displays text information corresponding to the audio notification regarding the image processing by the audio device and / or an information code that can be converted into text information corresponding to the audio notification, in synchronization with the audio notification.

[0006] The program relating to the second aspect causes an image processing device capable of cooperating with an audio device having an audio notification function to perform the steps of performing image processing using an image processing unit and displaying information related to the image processing on a display unit, wherein the displaying step includes a step of displaying text information corresponding to the audio notification regarding the image processing by the audio device and / or an information code that can be converted into text information corresponding to the audio notification, in synchronization with the audio notification.

[0007] The server according to the third aspect comprises a first communication unit that communicates with an audio device having an audio notification function, a second communication unit that communicates with an image processing device capable of operating in cooperation with the audio device, the image processing device having an image processing unit that performs image processing and a display unit that displays information related to the image processing, and a control unit that controls the image processing device via the second communication unit, and causes the display unit to display text information corresponding to the audio notification regarding the image processing by the audio device and / or an information code that can be converted into text information corresponding to the audio notification, in synchronization with the audio notification.

[0008] FIG. 1 is a schematic diagram showing the configuration of an image processing system to which an image processing device according to an embodiment is applied. FIG. 2 is a front view of the image processing device according to an embodiment. FIG. 3 is a functional block diagram of the image processing device according to an embodiment. FIG. 4 is a processing flow diagram of the image processing system to which an image processing device according to an embodiment is applied. FIG. 5 is a schematic diagram showing character information displayed on a display of the image processing device according to an embodiment. FIG. 6(A) is a schematic diagram showing how character information is scrollably displayed on the display of the image processing device according to an embodiment, where the first part of the character information is displayed and FIG. 6(B) is a schematic diagram showing how the second part of the character information is displayed. FIG. 7(A) is a schematic diagram showing how a two-dimensional code is displayed together with character information on the display of the image processing device according to an embodiment. FIG. 7(B) is a schematic diagram showing how detailed content of the character information corresponding to the two-dimensional code is displayed on a smartphone that reads the two-dimensional code of FIG. 7(A). FIG. 8(A) is a schematic diagram showing how only a two-dimensional code is displayed on the display of the image processing device according to an embodiment, and FIG. 8(B) is a schematic diagram showing how character information corresponding to the two-dimensional code is displayed on a smartphone that reads the two-dimensional code of FIG. 8(A). Fig. 9(A) is a schematic diagram showing how selectable text information is displayed on a display of an image processing device according to an embodiment, and Fig. 9(B) is a schematic diagram showing how related information related to the selected text information is displayed on a display. Fig. 10 is a schematic diagram showing how user answer candidates for a voice notification are displayed on a display of an image processing device according to an embodiment. Fig. 11 is a functional block diagram of a server according to a modified example.

[0009] (Summary of the embodiment) Information such as selectable options presented by a smart speaker via voice may be difficult to hear due to ambient noise, or may be missed while concentrating on other tasks, making it difficult to accurately understand the information presented by the smart speaker.

[0010] Therefore, the present disclosure provides an image processing device that allows a user to accurately understand information presented by an audio device without failing to receive it.

[0011] The image processing device of the embodiment is an image processing device capable of operating in cooperation with an audio device having audio notification and communication functions, and is equipped with an image processing unit that performs image processing, a display unit that displays information related to the image processing, and a control unit that controls the operation of the image processing unit and the display unit, and the control unit displays, on the display unit, text information corresponding to the audio notification regarding the image processing by the audio device and / or an information code that can be converted into text information corresponding to the audio notification, in synchronization with the audio notification.

[0012] In the image processing device of the present invention, the display of the image processing device displays text information corresponding to the voice notification from the audio device in synchronization with the voice notification, allowing information related to printing operations to be recognized and accurately understood both visually and audibly.

[0013] An image processing apparatus according to an embodiment will be described below. (Configuration of Image Processing System) Fig. 1 is a schematic diagram showing the configuration of an image processing system 1 to which an image processing apparatus 2 according to an embodiment is applied.

[0014] As shown in FIG. 1, the image processing system 1 is configured such that an image processing device 2 and an audio device 3 used together therewith are connected to a network N via an access point 4, and are connected to a server 5 via this network N.

[0015] In this embodiment, the image processing device 2 is capable of operating in cooperation with the audio device 3, and has a function of displaying text information corresponding to audio notifications from the audio device 3, and a printer function capable of printing images. However, the present invention is not limited to this, and the image processing device 2 may have various functions other than the printer function, such as a scanner function for reading and saving images as data, a copy function for printing the read image using a printer mechanism, or a facsimile function for sending the read image to another device, or may be a device without a printer function.

[0016] The audio device 3 is a device with audio pickup, pronunciation and communication functions, such as a smart speaker, smartphone, smart watch, tablet terminal, etc., that picks up audio including operational instructions to various devices made by the user, converts it into audio information, and transmits it to the server 5, as well as receiving the audio information transmitted from the server 5 and pronouncing it.

[0017] The access point 4 is a device for connecting a plurality of devices to each other and to the network N, and is installed, for example, in the user's home.

[0018] The server 5 analyzes the audio information received from the audio device 3 via the network N, extracts the operation instructions contained in the audio information, and transmits operation instruction information corresponding to the extracted operation instructions to the target device such as the image processing device 2, thereby causing the device to perform the operation.

[0019] (Configuration of image processing device) Fig. 2 is a front view of the image processing device according to the embodiment. Fig. 3 is a functional block diagram of the image processing device according to the embodiment. As shown in Figs. 2 and 3, the image processing device 2 is capable of operating in cooperation with the audio device 3, and is configured to include an image processing unit 20, a display 30 as a display unit, an input unit 40, a communication unit 50, a control unit 60, and a storage unit 70.

[0020] In this embodiment, the image processing unit 20 is a printer mechanism having an image printing function. Under the control of the control unit 60, the image processing unit 20 prints an image based on data received from the outside by the communication unit 50, an image based on data stored in a recording medium (not shown) connected to the image processing device 2, or an image based on data stored in the storage unit 70.

[0021] In this embodiment, the image processing unit 20 is a printer mechanism having an image printing function, but the present invention is not limited to this, and the image processing unit 20 may have various mechanisms other than a printer mechanism, such as a scanner function that reads images and stores them as data in the memory unit 70, or a copy function that prints the read images using a printer mechanism.

[0022] The display 30 displays text information and information codes related to the printing function. The information codes may be one-dimensional codes such as barcodes or two-dimensional codes such as QR (Quick Response) Codes (registered trademark). The display 30 also displays various information, such as information related to the settings and adjustments of the image processing device 2, as well as images related thereto. In this embodiment, the display 30 is a touch panel display formed by overlapping a display and a touch panel, and performs part of the function of the input unit 40. The display 30 is configured as a liquid crystal display or an organic electroluminescence (EL) display. These displays have a relatively large number of regularly arranged pixels and can display images including arbitrary shapes based on image data. The display 30 may also be capable of displaying color images.

[0023] The input unit 40 is configured to accept input operations from the user, and is composed of the display 30 as a touch panel display and one or more physical buttons 40b, as shown in Fig. 2. When a touch operation is performed on the display 30, the input unit 40 detects the position of the touch operation and outputs the detection result to the control unit 60. The input unit 40 may include an A / D converter that performs analog-to-digital (A / D) conversion of the output of the display 30.

[0024] The communication unit 50 performs image data communication under the control of the control unit 60. Specifically, the communication unit 50 realizes communication between the image processing device 2 and other devices. In this embodiment, the other devices include the audio device 3, a personal computer (PC), a mobile terminal (smartphone, etc.), another image processing device, and a server 5. Communication may be performed directly with the other devices or indirectly via a network N. Examples of networks include a telephone network, the Internet, a private network, and a LAN (Local Area Network). Furthermore, communication may be wired communication or wireless communication.

[0025] The control unit 60 includes one or more processors and controls the entire image processing device 2. The control unit 60 executes various processes by executing control programs stored in the storage unit 70. The control unit 60 controls the operations of the image processing unit 20, the display 30, the input unit 40, and the communication unit 50. For example, when the control unit 60 receives an operation (instruction) from a user via the input unit 40, it executes a process according to the operation content. The input unit 40 displays various operation screens on the display 30 in accordance with the instructions from the control unit 60. The control unit 60 may also include a logic circuit or the like configured to perform only certain operations.

[0026] The storage unit 70 includes various types of memory, such as a read-only memory (ROM), a random access memory (RAM), and an auxiliary storage device. The combination of the control unit 60 and the storage unit 70 may be considered as a computer. The programs executed by the control unit 60 are stored in the ROM of the storage unit 70 and / or the auxiliary storage device, for example.

[0027] The image processing unit 20, the display 30, the input unit 40, the communication unit 50, the control unit 60, and the storage unit 70 are connected by, for example, a bus. In Fig. 3, all the components are connected to a single bus BS, but multiple buses may be provided. For example, an address bus, a data bus, and a control bus may be provided.

[0028] (Example of Operation of Image Processing System) Next, a description will be given of the operation of the image processing system 1 having the above-described configuration. Fig. 4 is a process flow diagram of the image processing system 1 to which the image processing device 2 according to the embodiment is applied.

[0029] As shown in FIG. 4, in the image processing system 1, the audio device 3 first picks up the voice uttered by the user, including an operation instruction to the image processing device 2, converts it into audio information (step S1), and transmits the audio information to the server 5 (step S2).

[0030] Next, the server 5 analyzes the received voice information and extracts the operation instruction contained in the voice information (step S3).

[0031] The server 5 also transmits notification information (voice notification information) to the voice device 3 for making a voice notification related to the operation instruction (step S4).

[0032] Furthermore, the server 5 transmits notification information (text notification information) for displaying text information related to the operation instruction to the image processing device 2 simultaneously with or after a predetermined time from the voice notification to the audio device 3 (step S5). The text notification information may include information (e.g., two-dimensional code information) for displaying an information code (e.g., two-dimensional code) corresponding to the text notification information on the image processing device 2. Furthermore, the server 5 may transmit only two-dimensional code information to the image processing device 2 instead of the text notification information.

[0033] If the operation instruction extracted in step S3 can be immediately executed, the server 5 transmits operation instruction information corresponding to the operation instruction to the image processing device 2 and executes steps S4 and S5. On the other hand, if it is necessary to give a predetermined notice to the user before executing the operation instruction extracted in step S3, for example, when there are multiple options for the operation instruction, the server 5 executes steps S4 and S5 without transmitting operation instruction information to the image processing device 2.

[0034] Next, the audio device 3 performs audio notification based on the received audio notification information (step S6).

[0035] Furthermore, the image processing device 2 displays text information corresponding to the voice notification on the display 30 based on the received text notification information (step S7). The text information is a text version of the voice included in the voice notification.

[0036] When the text notification information includes information to be displayed on the image processing device 2 (for example, two-dimensional code information), the image processing device 2 displays, for example, a two-dimensional code together with the text information on the display 30. Furthermore, when the image processing device 2 receives, for example, only two-dimensional code information, it displays the two-dimensional code on the display 30.

[0037] The display of text information (and / or information code) on the display 30 by the image processing device 2 is performed in synchronization with the audio notification by the audio device 3 under the control of the control unit 60. Here, the synchronization of text information and audio notification is a concept including the fact that at least a portion of the display time of the text information is included in the execution time of the audio notification by the audio device 3, and that the display time of the text information and the execution time of the audio notification are continuous.

[0038] Specifically, synchronization includes (i) the display time of the text information and the execution time of the voice notification being the same, (ii) the display time of the text information and the execution time of the voice notification partially overlapping, and (iii) the display time of the text information and the execution time of the voice notification being different times but occurring consecutively.

[0039] In the case of (ii) of synchronization, the display of the text information may be performed simultaneously with the start of the execution of the audio notification, or may be started after a predetermined time has elapsed since the start of the execution of the audio notification and before the audio notification ends. Furthermore, in the case of (ii) above, the display of the text information may be started before the audio notification. Furthermore, in the case of (ii) above, the display of the text information may be completed within the execution time of the audio notification, or may be completed after a predetermined time has elapsed after the execution time of the audio notification has elapsed.

[0040] The display time of the character information (and / or information code) by the image processing device 2 is determined based on information (display time information) related to the display time included in the character notification information (and / or, for example, two-dimensional code information) from the server 5. The display time information may include only the duration (for example, 20 seconds) of the display time of the character information (and / or information code), and the image processing device 2 may display the character information (and / or information code) for that duration immediately upon receiving the character notification information (and / or, for example, two-dimensional code information) from the server 5. Alternatively, the display time information may include the start and end times of the display, and the image processing device 2 may display the character information (and / or information code) based on the start and end times.

[0041] (Example of Display by Image Processing Device) Next, a description will be given of display of characters and information codes (two-dimensional codes, etc.) by the image processing device 2. Fig. 5 is a schematic diagram showing character information T displayed on the display 30 of the image processing device 2 according to the embodiment.

[0042] 5, text information corresponding to the audio notification by the audio device 3 is displayed on the display 30 of the image processing device 2. In the example shown in Fig. 5, the audio device 3 issues a voice saying "Printing a coloring book" as the audio notification, and the display 30 of the image processing device 2 displays text information T saying "Printing a coloring book" corresponding to the audio notification.

[0043] Figure 6 (A) is a schematic diagram showing how character information T is displayed in a scrollable manner on the display 30 of the image processing device 2 of the embodiment, with the first part Ta of the character information T being displayed, and Figure 6 (B) is a schematic diagram showing how the second part Tb of the character information T is displayed.

[0044] The text information T may be displayed in its entirety on the display 30 at the same time, or if the display 30 is small and the entire text information T cannot fit on the display 30, the text information T may be displayed in a scrollable manner, as shown in Figures 6(A) and 6(B). In this case, for example, a scroll bar SB is displayed in an operable manner near the right edge of the display 30, and initially only the beginning portion Ta is displayed, with the subsequent latter portion Tb hidden. The user can then touch and slide the scroll bar SB to scroll the text information T from the beginning portion Ta to the latter portion Tb, or vice versa, thereby gradually switching the display.

[0045] By displaying the character information T in a scrollable manner in this way, it is possible to deal with cases where the entire character information T does not fit on the display 30 .

[0046] Furthermore, if the entire text information T does not fit on the display 30, a page-switching display, in which the display switches page by page, may be used instead of a scrolling display. Specifically, initially, only the page of the beginning portion Ta is displayed, and the pages of the latter portion Tb are hidden. Then, when the user touches and slides the scroll bar SB, the display switches from the page of the beginning portion Ta to the page of the latter portion Tb.

[0047] By displaying the text information T in a page-switchable manner in this way, it is possible to deal with cases where the entire text information T does not fit on the display 30 .

[0048] Figure 7 (A) is a schematic diagram showing how a two-dimensional code B is displayed together with character information T on the display 30 of the image processing device 2 according to the embodiment, and Figure 7 (B) is a schematic diagram showing how detailed content TS of the character information T corresponding to the two-dimensional code B is displayed on a smartphone S that has read the two-dimensional code B in Figure 7 (A).

[0049] 7A, the content of the text information T may be simplified and a two-dimensional code B may be displayed together with the text information T. In this example, the text information T is abbreviated and the message "Which would you like to print, an adult coloring book or a child's coloring book?" is displayed on the display 30 together with "...", which indicates that the previous content has been omitted.

[0050] Then, when the user reads this two-dimensional code B, for example, with the camera of the smartphone S, detailed content TS of the text information T may be displayed on the screen DS of the smartphone S.

[0051] Here, as the detailed content TS, the following text is displayed on the display 30, which reflects the content omitted by "..." in Figure 7 (A), and states: "There are coloring books for adults and coloring books for children. Adult coloring books have complex designs that can be enjoyed for a long time. Adult coloring books include landscapes and geometric patterns. Children's coloring books have simple designs that can be enjoyed in a short time. Children's coloring books include vehicles and characters. Which would you like to print, the adult coloring book or the child's coloring book?"

[0052] In this manner, even by displaying the two-dimensional code B together with the character information T, it is possible to deal with cases where the detailed content TS of the character information T cannot all fit on the display 30.

[0053] Figure 8 (A) is a schematic diagram showing how only a two-dimensional code B is displayed on the display 30 of an image processing device 2 according to an embodiment, and Figure 8 (B) is a schematic diagram showing how text information T corresponding to the two-dimensional code B is displayed on a smartphone S that has read the two-dimensional code B in Figure 8 (A).

[0054] As shown in Figure 8 (A), if the display 30 of the image processing device 2 is small and it is difficult to display the content of the character information T in a sufficiently identifiable manner, only the two-dimensional code B may be displayed on the display 30.

[0055] Then, when the user reads the two-dimensional code B with the camera of the smartphone S, text information T may be displayed on the screen DS of the smartphone S. Here, as the text information T corresponding to the two-dimensional code B, the following text is displayed on the display 30: "There are coloring books for adults and coloring books for children. Adult coloring books have complex designs that can be enjoyed for a long time. Adult coloring books include landscapes and geometric patterns. Children's coloring books have simple designs that can be enjoyed in a short time. Children's coloring books include vehicles and characters. Which would you like to print, the adult coloring book or the child's coloring book?"

[0056] In this way, by displaying only the two-dimensional code B on the display 30 and reading the text information T corresponding to the two-dimensional code B with an external smartphone S or the like and displaying it thereon, it becomes possible to accommodate even small displays 30.

[0057] Figure 9 (A) is a schematic diagram showing how selectable character information TC is displayed on the display 30 of the image processing device 2 of the embodiment, and Figure 9 (B) is a schematic diagram showing how related information T2 related to the selected character information TC is displayed on the display 30.

[0058] As shown in Fig. 9A, character information T displayed on the touch panel display 30 may include character information TC that can be selected by a touch operation. In Fig. 9A, character information T saying "Print a coloring book" is displayed, and the "coloring book" part of this character information T is selectable character information TC.

[0059] To make it easier for the user to recognize that the selectable character information TC is selectable, it is preferable that the selectable character information TC be displayed in a manner different from that of character information that is not selectable character information TC, for example by changing the color of the background part of the selectable character information TC from that of the other character parts, or by changing the color or thickness of the font, or by displaying the font in a blinking manner.

[0060] Furthermore, the selectable character information TC is not limited to words contained in the character information, but may be at least one of a sentence and a paragraph, and the selectable character information TC can be displayed in various forms.

[0061] When the user touches selectable text information TC, the operation information is transmitted from the image processing device 2 to the server 5. The server 5 generates text notification information for displaying related information T2 related to the received operation information and transmits it to the image processing device 2, and also generates and transmits audio notification information corresponding to the text notification information to the audio device 3.

[0062] The control unit 60 of the image processing device 2 that has received the text notification information causes the display 30 to display related information T2 corresponding to the text notification information.

[0063] In the example shown in FIG. 9(B), the display 30 of the image processing device 2 that has received the text notification information displays the following related information T2: "There are four types of 'coloring books': Landscapes of cities and mountain villages, etc.; Plants such as flowers and vegetables; Animals such as insects and birds; and Complex geometric patterns."

[0064] The audio device 3 also issues an audio notification corresponding to the related information T2.

[0065] In addition, the related information T2 is not limited to information that embodies the content of the selectable character information TC, but may also include any character information that is different from the selectable character information TC but is associated with the selectable character information TC.

[0066] In this way, by including selectable character information TC in character information T, the amount of information displayed on display 30 at one time can be reduced, making it possible to accommodate small displays 30 and presenting information to the user in an easy-to-understand manner.

[0067] FIG. 10 is a schematic diagram showing how candidate answers TB1 and TB2 of a user to a voice notification are displayed on the display 30 of the image processing device 2 according to the embodiment.

[0068] As shown in Figure 10, when the voice notification by the voice device 3 includes multiple options, the display 30 of the image processing device 2 may display the user's answer candidates TB1 and TB2 for the options in the text information T.

[0069] 10, the audio device 3 issues a voice notification asking, "There are coloring books for adults and children. Which one would you like to print?" The display 30 of the image processing device 2 displays, as text information T corresponding to this voice notification, "Which coloring book would you like to print?", along with two objects (answer candidates TB1 and TB2) as answer candidates: "For adults" indicating a coloring book for adults and "For children" indicating a coloring book for children.

[0070] Then, when the user selects one of the two answer candidates TB1, TB2 by operating the display 30, which is a touch panel display, or by voice input via the audio device 3, selection information corresponding to the selected answer candidate TB1, TB2 is transmitted from the image processing device 2 or the audio device 3 to the server 5.

[0071] Then, the server 5 sends an instruction to print a coloring picture corresponding to the selected answer candidates TB1, TB2 as an operation instruction corresponding to the selection information to the image processing device 2, and the image processing device 2 prints the coloring picture.

[0072] In this way, by displaying answer candidates TB1 and TB2 within the character information T, it is possible to present information to the user in an easy-to-understand manner, and it is also possible for the user to easily perform the operation of selecting answer candidates TB1 and TB2.

[0073] In addition, when the selection of answer candidates TB1 and TB2 is made by voice input to the audio device 3, information regarding the selection is sent to the server 5, and the server 5 sends to the image processing device 2 an operational instruction to terminate the display of answer candidates TB1 and TB2 and / or an invalidation instruction to invalidate the selection operation on the display 30.

[0074] Then, based on the operation instruction and / or the invalidation instruction, the control unit 60 causes the display 30 to end the display of the answer candidates TB1 and TB2 and / or invalidates the selection operation on the display 30. This makes it possible to prevent duplicate selection operations from being performed.

[0075] According to the image processing device 2 of this embodiment described above, at least one of the text information corresponding to the voice notification by the audio device 3 and the two-dimensional code that can be converted into text information corresponding to the voice notification is displayed so that at least a portion of the display time of the text information and the two-dimensional code is included in the execution time of the voice notification. Therefore, even if the user has difficulty hearing the voice notification due to surrounding noise or misses it because they are concentrating on other work, the user can check the content of the voice notification on the display 30 of the image processing device 2, and can accurately understand it without missing it.

[0076] It should be noted that various modifications can be made to the image processing device 2 according to the above-described embodiment.

[0077] For example, the control unit 60 may cause the display 30 to hide text information corresponding to a previously given voice notification while the voice device 3 is giving a voice notification. That is, before the next voice notification by the voice device 3 starts, the text information corresponding to the previously given voice notification is erased from the display 30. This prevents the old text information corresponding to the old voice notification from being displayed, making it possible to accurately understand the text information corresponding to the content of the currently given voice notification.

[0078] Furthermore, in the above-described embodiment, the control unit 60 displays the text information T on the display 30 in a manner that allows scrolling or page switching, thereby making it possible to deal with cases where the entire text information T does not fit on the display 30. However, in addition to this aspect, the present invention may also be configured in a manner in which, for example, the control unit 60 displays a portion of the text information on the display 30 and hides the other portion, and sequentially displays the other portion in response to a voice notification from the audio device 3.

[0079] Specifically, the control unit 60 may cause the display 30 to sequentially display the text information T from the beginning to the end as the audio notification by the audio device 3 progresses. At this time, the control unit 60 may cause the display 30 to switch the display range of the text information T from the beginning to the end. The control unit 60 may also cause the display 30 to switch the display of the text information T sentence by sentence from the beginning to the end. Alternatively, the control unit 60 may cause the display 30 to display the text information T character by character from the beginning to the end. These modes also make it possible to deal with cases where the entire text information T does not fit on the display 30.

[0080] Furthermore, although the image processing device 2 and the audio device 3 are separate devices in the above-described embodiment, they may be integrated into one device (audio-image processing device). That is, in the audio-image processing device, the audio notification and text information or two-dimensional code are displayed on the display in synchronization with the audio notification. Even in this manner, even if the user has difficulty hearing the audio notification due to ambient noise or misses it while concentrating on other tasks, the user can still check the content of the audio notification on the display of the audio-image processing device, allowing the user to accurately understand the notification without missing it.

[0081] (Modified Example) Although an example has been described in which the image processing device 2 has "a control unit 60 that causes the display unit (display 30) to display, in synchronization with the voice notification, character information T corresponding to the voice notification related to image processing by the voice device 3 and / or a two-dimensional code B that can be converted into character information T corresponding to the voice notification," such a control unit 60 may be provided in the server 5. In other words, the server 5 may be configured to remotely control the image processing device 2.

[0082] 11 is a functional block diagram of a server 5A according to a modified example. As shown in FIG. 11, the server 5A according to the modified example is configured to include a first communication unit 501, a second communication unit 502, and a control unit 503.

[0083] The first communication unit 501 performs data communication with an audio device 3 (see FIG. 1 ) having sound generation, sound collection, and communication functions. The communication may be performed directly with the audio device 3 or indirectly via a network N. Examples of the network include a telephone network, the Internet, a private network, and a LAN (Local Area Network). The communication may be wired or wireless.

[0084] The second communication unit 502 performs data communication with the image processing device 2 (see FIGS. 1 and 3 ), which is an image processing device 2 capable of operating in cooperation with the audio device 3 and has an image processing unit 20 that performs image processing and a display unit 30 that displays information related to the image processing. This communication may also be performed directly with the image processing device 2, or indirectly via the network N.

[0085] The control unit 503 controls the image processing device 2 via the second communication unit 502. Specifically, the control unit 503 causes the display unit 30 of the image processing device 2 to display, in synchronization with the voice notification, character information T corresponding to the voice notification regarding image processing by the voice device 3 and / or a two-dimensional code B that can be converted into character information corresponding to the voice notification.

[0086] The synchronization referred to here also includes (i) the display time of the text information and the execution time of the voice notification being the same, (ii) the display time of the text information and the execution time of the voice notification partially overlapping, and (iii) the display time of the text information and the execution time of the voice notification being different times but occurring consecutively.

[0087] Then, this server 5A performs the same operations as the control operations performed by the control unit 60 described in the above embodiment.

[0088] With this configuration, even if the user has difficulty hearing the voice notification due to surrounding noise or misses it because they are concentrating on other work, they can check the content of the voice notification on the display 30 of the image processing device 2, so they can accurately understand the notification without missing it.

[0089] Other Embodiments The operational flows in the above-described embodiments do not necessarily have to be executed in chronological order according to the order depicted in the flow diagrams. For example, steps in the operations may be executed in an order different from that depicted in the flow diagrams, or may be executed in parallel. Furthermore, some steps in the operations may be deleted, or additional steps may be added to the process.

[0090] A program may be provided that causes a computer (image processing device) to execute the operations according to the above-described embodiments. The program may be recorded on a computer-readable recording medium. Using the computer-readable recording medium, the program can be installed on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transitory print medium. The non-transitory storage medium is not particularly limited, and may be, for example, a storage medium such as a CD-ROM or a DVD-ROM.

[0091] As used in this disclosure, the terms "based on" and "depending on / in response to" do not mean "based only on" or "depending only on," unless expressly stated otherwise. The term "based on" means both "based only on" and "based at least in part on." Similarly, the term "depending on" means both "depending only on" and "depending at least in part on." Furthermore, the terms "include," "comprise," and variations thereof do not mean including only the listed items, but may mean including only the listed items or may include additional items in addition to the listed items. Furthermore, the term "or," as used in this disclosure, is not intended to mean an exclusive or. Furthermore, any reference to elements using designations such as "first," "second," etc., as used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used herein as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed therein or that the first element must precede the second element in some way. In this disclosure, where articles are added by translation, such as a, an, and the in English, these articles shall include the plural unless the context clearly indicates otherwise.

[0092] The above describes the embodiments in detail with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes can be made within the scope that does not deviate from the gist of the invention.

[0093] (Additional Notes) Additional notes will be given regarding the features of the above-described embodiment.

[0094] (1) An image processing device capable of cooperating with an audio device having an audio notification function, comprising: an image processing unit that performs image processing; and a display unit that displays information related to the image processing, wherein the display unit displays text information corresponding to the audio notification regarding the image processing by the audio device and / or an information code that can be converted into text information corresponding to the audio notification, in synchronization with the audio notification.

[0095] (2) The image processing device according to (1), wherein the display unit displays the character information and / or the information code so that at least a portion of the display time of the character information and / or the two-dimensional code is included in the execution time of the voice notification.

[0096] (3) The image processing device according to (1), wherein the display unit, while the audio device is executing the audio notification, does not display text information and / or the information code corresponding to an audio notification executed before the audio notification.

[0097] (4) The image processing device according to (1), wherein when a part of the character information is displayed and the other part is not displayed, the display unit is configured to enable the character information of the other part to be confirmed by scrolling or switching pages.

[0098] (5) The image processing device according to (1), wherein the display unit displays at least a portion of the character information in a user-selectable state, and when a portion of the character information is selected, displays related information related to the selected character information.

[0099] (6) The image processing device according to (5), wherein the selectable character information is at least one of a word, a sentence, and a paragraph included in the character information.

[0100] (7) The image processing device according to (5), wherein the selectable character information is displayed in a manner different from that of character information that is not the selectable character information.

[0101] (8) The image processing device according to (5), further comprising an input unit that accepts an input operation from a user, wherein the input unit accepts an operation from the user to select the selectable character information.

[0102] (9) The image processing device according to (5), wherein the related information is information that embodies the content of the selectable character information or information that is different from the selectable character information but is associated with the selectable character information.

[0103] (10) The image processing device according to (1), wherein the display unit sequentially displays the text information from the beginning to the end as the audio notification by the audio device progresses.

[0104] (11) The image processing device according to (10), wherein the sequential display of the character information is performed by switching the display range from the beginning to the end of the character information, by switching the display on a sentence-by-sentence basis, or by displaying each character one by one.

[0105] (12) The image processing device according to (1), further comprising an input unit that accepts an input operation from a user, wherein the display unit displays answer candidates for the user in response to the voice notification, and the input unit accepts an operation from the user to select at least one of the answer candidates.

[0106] (13) The image processing device according to (12), wherein the display unit displays a plurality of objects indicating the answer candidates in response to the audio device outputting a question to the user by the audio notification.

[0107] (14) The image processing device according to (12), wherein when a selection of the answer candidate displayed on the display unit is made by voice input to the voice device, the display unit terminates the display of the answer candidate, and / or the input unit invalidates the user's answer input.

[0108] (15) The image processing device according to any one of (1) to (14), wherein the text information corresponding to the voice notification is a text version of the voice included in the voice notification.

[0109] (16) A program that causes an image processing device that can be linked with an audio device having an audio notification function to execute the steps of: performing image processing using an image processing unit; and displaying information related to the image processing on a display unit, wherein the displaying step includes a step of displaying text information corresponding to the audio notification regarding the image processing by the audio device and / or an information code that can be converted into text information corresponding to the audio notification, in synchronization with the audio notification.

[0110] (17) A server comprising: a first communication unit that communicates with an audio device having an audio notification function; a second communication unit that communicates with an image processing device capable of operating in cooperation with the audio device, the image processing device having an image processing unit that performs image processing and a display unit that displays information related to the image processing; and a control unit that controls the image processing device via the second communication unit, wherein the server causes the display unit to display text information corresponding to the audio notification regarding the image processing by the audio device and / or an information code that can be converted into text information corresponding to the audio notification, in synchronization with the audio notification.

[0111] 1: Image processing system 2: Image processing device 3: Audio device 4: Access point 5, 5A: Server 30: Display 40: Input unit 40b: Physical button 50: Communication unit 60: Control unit 70: Storage unit 501: First communication unit 502: Second communication unit 503: Control unit B: Two-dimensional code N: Network SB: Scroll bar T: Text information Ta: Opening part Tb: Second half TB1, TB2: Answer candidates TS: Detailed content

Claims

1. An image processing apparatus capable of cooperating with an audio device having an audio notification function, comprising: an image processing unit that performs image processing; a display unit that displays information related to the image processing, wherein the display unit displays character information corresponding to the audio notification related to the image processing by the audio device and / or an information code convertible into the character information corresponding to the audio notification in synchronization with the audio notification.

2. The image processing apparatus according to claim 1, wherein the display unit displays the character information and / or the information code such that at least a part of the display time of the character information and / or the information code is included in the execution time of the audio notification.

3. The image processing apparatus according to claim 1, wherein during the execution of the audio notification by the audio device, the display unit makes the character information and / or the information code corresponding to the audio notification performed before the current audio notification non-displayed.

4. The image processing apparatus according to claim 1, wherein when a part of the character information is displayed and the other part is non-displayed, the display unit enables the user to confirm the character information of the other part by scroll display or page switching.

5. The image processing apparatus according to claim 1, wherein the display unit displays at least a part of the character information in a selectable state by the user, and when a part of the character information is selected, the display unit displays related information related to the selected character information.

6. The image processing apparatus according to claim 5, wherein the selectable character information is at least one of a word, a sentence, and a paragraph included in the character information.

7. The image processing apparatus according to claim 5, wherein the selectable character information is displayed in a manner different from that of the character information that is not the selectable character information.

8. The image processing apparatus according to claim 5, further comprising an input unit that receives a user input operation, wherein the input unit receives from the user an operation of selecting the selectable character information.

9. The image processing apparatus according to claim 5, wherein the related information is information that concretizes the content of the selectable character information or character information that is different from the selectable character information but is associated with the selectable character information.

10. The image processing apparatus according to claim 1, wherein the display unit sequentially displays the character information from the beginning to the end as the audio notification by the audio device progresses.

11. The sequential display of the text information is performed by switching the display range from the beginning to the end of the text information, by switching the display at the sentence unit, or by displaying one character at a time. The image processing apparatus according to claim 10.

12. The image processing apparatus according to claim 1, further comprising an input unit that receives a user's input operation, wherein the display unit displays candidate answers of the user to the voice notification, and the input unit receives from the user an operation of selecting at least one of the candidate answers.

13. The image processing apparatus according to claim 12, wherein the display unit displays a plurality of objects indicating the candidate answers in response to the voice device outputting a question to the user by the voice notification.

14. When the selection of the candidate answer displayed on the display unit is made by voice input to the voice device, the display unit ends the display of the candidate answer, and / or the input unit invalidates the user's answer input. The image processing apparatus according to claim 12.

15. The character information corresponding to the voice notification is obtained by converting the voice included in the voice notification into characters. The image processing apparatus according to any one of claims 1 to 14.

16. A program for causing an image processing apparatus capable of cooperating with a voice device having a voice notification function to execute a step of performing image processing by an image processing unit and a step of displaying information related to the image processing on a display unit, wherein the displaying step includes a step of synchronously displaying character information corresponding to a voice notification regarding the image processing by the voice device and / or an information code convertible into the character information corresponding to the voice notification.

17. A server comprising: a first communication unit that communicates with a voice device having a voice notification function; a second communication unit that communicates with the image processing apparatus that is an image processing apparatus capable of operating in cooperation with the voice device and has an image processing unit that performs image processing and a display unit that displays information related to the image processing; and a control unit that controls the image processing apparatus via the second communication unit, and causes the display unit to synchronously display character information corresponding to a voice notification regarding the image processing by the voice device and / or an information code convertible into the character information corresponding to the voice notification.

Images