Notification system, control method therefor, and program

Abstract

To provide a notification system that is capable of sending a message to a person who is highly related to the message when notifying of the message, a control method and a program of the notification system.SOLUTION: A notification system 100 includes an image pickup unit 106 that picks up an image including a person, a system control program 700 as acquisition means that acquires attribute information about the attribute of a person in the image based on the image picked up by the image pickup unit 106, and a speaker 310 as notification means that notifies of a message about a prescribed event. The notification means switches contents of the message according to the attribute information acquired by the acquisition means when notifying of the message.SELECTED DRAWING: Figure 15

Description

Technical Field

[0001] The present invention relates to a notification system, a control method for the notification system, and a program.

Background Art

[0002] Conventionally, among image forming apparatuses (image processing apparatuses) such as MFPs (Multifunction Peripherals / Printers / Products), there are apparatuses capable of notifying messages to users who use the apparatuses. As one of such message notifications, there is a method of notifying a message by voice from a speaker. In the apparatus described in Patent Document 1, when a problem such as a trouble of the apparatus occurs, a message requesting maintenance is automatically generated internally. Then, a message requesting maintenance is notified by voice to unspecified users who approach the apparatus.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the apparatus described in Patent Document 1, when a voice message is notified to an unspecified user, the user who receives the voice message is not necessarily a user who can respond according to the message or a user who can actually respond. For example, when a problem such as a failure of a conveyance roller that requires specialized knowledge occurs, even if a message regarding its recovery is notified to a general user, problems such as it being difficult for the user to respond or causing confusion to the user may occur.

[0005] This invention has been made in view of the above-mentioned problems. The object of this invention is to provide a notification system, a control method for the notification system, and a program that can notify individuals who are highly relevant to a message when a message is notified. [Means for solving the problem]

[0006] To achieve the above objective, the notification system of the present invention is characterized by comprising: an imaging means for capturing an image including a person; an acquisition means for acquiring the role of the person in the image based on the image captured by the imaging means; a determination means for determining the content of a message about a predetermined event according to the role acquired by the acquisition means; and a notification means for notifying the message determined by the determination means. [Effects of the Invention]

[0007] According to the present invention, when a message is notified, it is possible to notify individuals who are highly relevant to that message. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the overall configuration of a notification system according to the first embodiment of the present invention. [Figure 2] Figure 1 is a block diagram showing the hardware configuration of the image forming apparatus. [Figure 3] Figure 1 is a block diagram showing the hardware configuration of the voice control device. [Figure 4] Figure 1 is a block diagram showing the hardware configuration of the cloud server. [Figure 5] This block diagram shows the hardware configuration of the imaging device shown in Figure 1. [Figure 6] This is a block diagram showing the configuration of the software (device control program) for an image forming apparatus. [Figure 7] This is a block diagram showing the configuration of the software (voice control program) for a voice control device. [Figure 8] This is a block diagram showing the configuration of the cloud server software (system control program). [Figure 9] This is a block diagram showing the configuration of the software (camera control program) for the imaging device. [Figure 10] This diagram shows a recovery procedure management table managed on a cloud server. [Figure 11A] This diagram shows a user-type message generation table managed on a cloud server. [Figure 11B] This diagram shows a user-type message generation table managed on a cloud server. [Figure 12] This is a sequence diagram showing the surrounding monitoring and control process performed between the camera control program and the system control program. [Figure 13] This sequence diagram shows the processing from device anomaly detection to voice notification, which takes place between the device control program, system control program, and voice control program. [Figure 14] This is a flowchart showing the surrounding area monitoring process performed by the camera control program. [Figure 15] This flowchart shows the user detection and identification process performed by the system control program. [Figure 16] This flowchart shows the device abnormality information notification process by the device control program. [Figure 17] This flowchart shows the process of generating messages for detected users and transmitting them via speech synthesis by the system control program. [Figure 18] Figure 17 is a flowchart showing the message generation process executed by the system control program at step S4005. [Figure 19] This is a flowchart showing the voice notification process by a voice control program. [Figure 20A] This is an overhead view showing an example of the arrangement of an image forming apparatus, a shooting apparatus, and a user being monitored. [Figure 20B] This is a diagram to explain facial recognition. [Figure 20C] It is a bird's-eye view showing an example of the arrangement of an image forming apparatus, a photographing apparatus, and a user to be monitored. [Figure 20D] It is a bird's-eye view showing an example of the arrangement of an image forming apparatus, a photographing apparatus, and a user to be monitored. [Figure 20E] It is a bird's-eye view showing an example of the arrangement of an image forming apparatus, a photographing apparatus, and a user to be monitored. [Figure 20F] It is a bird's-eye view showing an example of the arrangement of an image forming apparatus, a photographing apparatus, and a user to be monitored. [Figure 21] It is a diagram showing an example of a feature when determining a user type from an image captured by a photographing apparatus. [Figure 22] It is a flowchart showing the user-specific message generation process executed in S5002 of FIG. 18. [Figure 23] It is a block diagram showing the configuration of software (data conversion control program) of a cloud server included in the notification system according to the second embodiment of the present invention. [Figure 24] It is a sequence diagram showing the processes carried out among a camera control program, a device control program, an audio control program, and a data conversion control program. [Figure 25] It is a flowchart showing the audio message control process by a data conversion control program. [Figure 26] It is a diagram showing an example of a user data list used in a data conversion control program. [Figure 27] It is a diagram showing an example of a message generation table stored in a cloud server. [Figure 28] It is a bird's-eye view showing an example of the arrangement of an image forming apparatus, a photographing apparatus, and a user to be monitored. [Figure 29] It is a diagram showing an example of a feature when determining a user type from an image captured by a photographing apparatus.

Embodiments for Carrying Out the Invention

[0009] Embodiments of the present invention will be described in detail below with reference to the drawings. However, the configurations described in the following embodiments are merely illustrative, and the scope of the present invention is not limited by the configurations described in the embodiments. For example, each part constituting the present invention can be replaced with any configuration that can perform a similar function. In addition, any additional components may be added. Furthermore, any two or more configurations (features) from the embodiments can be combined.

[0010] <First Embodiment> The notification system of the present invention identifies the role of a person in an image based on the captured image, and determines the content of the message according to the identified role. In the first embodiment, the system determines the attribute of the person (user, administrator, etc.) as their role based on their clothing, and determines the content of the message based on this.

[0011] <Overall Structure> Figure 1 shows an example of the overall configuration of a notification system 100 according to a first embodiment of the present invention. As shown in Figure 1, the notification system 100 includes an image forming apparatus (image processing device) 101, a voice control device 102, a cloud server 103, a gateway 105, and a shooting device (imaging means) 106. In this notification system 100, the image forming apparatus 101, the voice control device 102, the shooting device 106, and the cloud server 103 are connected to each other so as to be able to communicate with each other via the gateway 105 and the network 104. The notification system 100 stores a program that causes a computer to execute the operation of each part and means of the notification system 100 (a control method for the notification system). This program may be stored collectively in one of the devices, for example, the image forming apparatus 101, the voice control device 102, the shooting device 106, or the cloud server 103, or it may be stored distributed among the devices.

[0012] The image forming apparatus 101 is an MFP (Multifunction Peripheral / Printer / Product) with multiple functions such as copying, scanning, printing, and faxing. In this embodiment, the image forming apparatus 101 is a detection target in which a malfunction occurs as a predetermined event, and that malfunction is detected. Note that the predetermined device to be used as a detection target is not limited to the image forming apparatus 101, but may be other information processing devices such as personal computers or mobile terminals. Also, the predetermined event is not limited to malfunctions, but may be, for example, a command to the device. Furthermore, the image forming apparatus 101 is not limited to a multifunction device, but may be a device with a single function, such as a printer or scanner.

[0013] The voice control device 102 records the user's voice in response to the user's voice operation start command and transmits the encoded voice data to the cloud server 103. The voice control device 102 is a voice input / output device that can communicate with the user via voice from, for example, a smart speaker or smartphone. In this embodiment, the voice control device 102 and the image forming apparatus 101 are configured as separate entities, i.e., they are independent configurations, but this is not limited to this. For example, at least one of the hardware and software functions that constitute the voice control device 102 may be included in the image forming apparatus 101.

[0014] The cloud server 103 is composed of one or more cloud servers and can provide cloud services such as user management using the notification system 100, voice and image recognition, and information processing for controlling the image forming apparatus 101. The cloud server 103 can recognize user voice data acquired by the voice control device 102 and image data captured by the image capture device 106, while referring to the user information managed by the cloud server 103. The cloud server 103 can also analyze device information such as malfunction information acquired from the image forming apparatus 101. Furthermore, the cloud server 103 generates a predetermined message according to the image recognition results and device information analysis results, and also synthesizes voice data for voice playback of the message content by the voice control device 102.

[0015] The imaging device 106 captures an image (imaging process) that includes a person (hereinafter sometimes referred to as "user") approaching the area surrounding the image forming apparatus 101, which is the area to be monitored. In this embodiment, there are three types of people: a general user, an administrator user, and a service technician. A general user is a person who uses the image forming apparatus 101. An administrator user is a supervisor who manages the status of the image forming apparatus 101. A service technician is a maintenance and inspection person who performs maintenance and inspection of the image forming apparatus 101. In this embodiment, the imaging device 106 is an external camera, i.e., a network camera, that is installed in a location separate from the image forming apparatus 101 and connectable via USB or LAN. However, it is not limited to this. For example, the imaging device 106 may be installed in the image forming apparatus 101, i.e., built into it.

[0016] Network 104 is connected to the image forming apparatus 101, the sound control device 102, and the imaging device 106, and is also connected to the cloud server 103 via the gateway 105. Network 104 can transfer various types of data, such as audio data acquired by the sound control device 102, image data captured by the imaging device 106, device information transmitted from the image forming apparatus 101, and audio data transmitted from the cloud server 103.

[0017] Gateway 105 is, for example, a wireless LAN router compliant with the IEEE 802.11 standard series. The "IEEE 802.11 standard series" refers to a series of standards belonging to IEEE 802.11, such as IEEE 802.11a and IEEE 802.11b. Gateway 105 is not limited to a wireless LAN router; it may also be a wired LAN router compliant with Ethernet standards such as 10BASE-T, 100BASE-T, or 1000BASE-T. Gateway 105 may also have the capability to operate according to other wireless communication methods or other wired communication methods.

[0018] <Hardware configuration of image forming apparatus> Figure 2 is a block diagram showing the hardware configuration of the image forming apparatus 101. As shown in Figure 2, the image forming apparatus 101 includes a controller unit 200, an operation panel 209, a print engine 211, and a scanner 213. The controller unit 200 includes a CPU (Central Processing Unit) 202. The controller unit 200 also includes RAM 203, ROM 204, storage 205, network interface 206, display controller 207, operation interface 208, print controller 210, and scan controller 212. These components in the controller unit 200 are connected to each other via a system bus 201 so that they can communicate with one another.

[0019] The CPU 202 controls the operation of the entire image forming apparatus 101. The CPU 202 reads control programs stored in the ROM 204 or storage 205 and performs various controls such as read control and print control. The RAM 203 is the main memory of the CPU 202 and is used as a temporary storage area for loading various control programs stored in the work area, ROM 204, and storage 205. The ROM 204 stores control programs that can be executed by the CPU 202. The storage 205 stores print data, image data, various programs, and various setting information.

[0020] In this embodiment, the image forming apparatus 101 uses one CPU 202 and one memory (RAM 203) to execute each process shown in the flowchart described later, but the execution configuration is not limited to this. For example, the image forming apparatus 101 can also execute each process shown in the flowchart described later by having multiple CPUs, RAM, ROM, and storage cooperate. In addition, some processes may be executed using hardware circuits such as ASICs (Application Specific Integrated Circuits) or FPGAs (Field-Programmable Gate Arrays).

[0021] The network interface 206 is an interface that allows the image forming apparatus 101 to communicate with external devices via a network. The image forming apparatus 101 also analyzes the print data received via the network interface 206 using a software module (PDL analysis unit) stored in the storage 205 or ROM 204. The PDL analysis unit generates data for printing by the print engine 211 based on the print data expressed in various types of page description languages. The display controller 207 is connected to the operation panel 209. In the image forming apparatus 101, a screen controlled by the operation interface 208 is displayed on the operation panel 209. By operating this screen, the user can obtain events corresponding to their operations via the display controller 207.

[0022] The print controller 210 is connected to the print engine 211. Image data to be printed is transferred to the print engine 211 via the print controller 210. The print engine 211 receives control commands and image data to be printed, and forms an image on the sheet based on this image data. The printing method of the print engine 211 may be electrophotographic or inkjet. If the printing method is electrophotographic, an electrostatic latent image is formed on the photoreceptor, then developed with toner, and the toner image is transferred to the sheet. The transferred toner image is then fixed to form an image. On the other hand, if the printing method is inkjet, ink is ejected to form an image on the sheet.

[0023] The scan controller 212 is connected to the scanner 213. The scanner 213 reads the image on the sheet and generates image data. This generated image data is stored in the storage 205. The image forming apparatus 101 can also form an image on the sheet using the image data generated by the scanner 213. The scanner 213 has a document feeder (not shown) and can read sheets placed on the document feeder while transporting them one by one.

[0024] <Hardware configuration of the voice control device> Figure 3 is a block diagram showing the hardware configuration of the voice control device 102. As shown in Figure 3, the voice control device 102 has a controller unit 300. The controller unit 300 includes a CPU 302, RAM 303, ROM 304, storage 305, network I / F 306, microphone I / F 307, audio controller 309, and display controller 311. These components are connected to each other via a system bus 301 so that they can communicate with one another. The voice control device 102 also further includes a microphone 308 as a voice input device, a speaker 310 as a voice output device, and an LED 312 as a notification device.

[0025] The CPU 302 is a central processing unit that controls the operation of the entire controller unit 300. The RAM 303 is volatile memory. The ROM 304 is non-volatile memory and stores the startup program for the CPU 302. The storage 305 is a storage device with a larger capacity compared to the RAM 303, and is composed of, for example, an SD card. The storage 305 stores the control program for the voice control device 102, which is executed by the controller unit 300.

[0026] Note that storage 305 is not limited to an SD card, but can be other storage devices with equivalent functionality to an SD card, such as flash ROM. When the voice control device 102 is powered on or otherwise started up, the CPU 302 executes a startup program stored in ROM 304. This startup program reads the control program stored in storage 305 and loads it onto RAM 303. After executing the startup program, the CPU 302 executes the control program loaded onto RAM 303 to control the operation of the entire controller unit 300. The CPU 302 also stores and reads / writes data used when executing the control program onto RAM 303. Storage 305 can also store various settings necessary when executing the control program. The CPU 302 can communicate with other devices on the network 104 via the network I / F 306.

[0027] The network interface 306 includes circuits and antennas for communication in accordance with the IEEE 802.11 standard series. Note that communication via the network interface 306 is not limited to wireless communication; for example, it may also use a wired communication method compliant with the Ethernet standard. The microphone interface 307 is connected to the microphone 308 and converts the user's voice input via the microphone 308 into encoded audio data. This audio data is stored in the RAM 303 according to instructions from the CPU 302.

[0028] In this embodiment, the microphone 308 is, for example, a small MEMS microphone mounted on a smartphone, but is not limited to this. Furthermore, while one or more microphones 308 may be installed, it is preferable to have three or more at predetermined locations. When three or more microphones 308 are installed, the direction of the sound emitted by the user can be detected. The audio controller 309 is connected to the speaker 310 and, in accordance with instructions from the CPU 302, converts the audio data into an analog audio signal and outputs the sound through the speaker 310.

[0029] Speaker 310 plays a response sound indicating that the voice control device 102 is responding, as well as voice synthesized by the cloud server 103. Speaker 310 is a general-purpose device for playing voice. Display controller 311 is connected to LED 312 and controls the display of LED 312 according to instructions from CPU 302. Here, display controller 311 mainly controls the lighting of the LED to indicate that the voice control device 102 is correctly inputting the user's voice. LED 312 is, for example, a blue LED that is visible to the user. LED 312 is a general-purpose device. Alternatively, a display device capable of displaying characters, drawings, etc., may be used instead of LED 312.

[0030] <Cloud Server Hardware Configuration> Figure 4 is a block diagram showing the hardware configuration of the cloud server 103. As shown in Figure 4, the cloud server 103 has a controller unit 400. The controller unit 400 has a CPU 402, RAM 403, ROM 404, storage 405, and network I / F 406, which are connected to each other via a system bus 401 so that they can communicate with one another.

[0031] The CPU 402 is a central processing unit that controls the operation of the entire controller unit 400. The RAM 403 is volatile memory. The ROM 404 is non-volatile memory and stores the startup program for the CPU 402. The storage 405 is a storage device with a larger capacity compared to the RAM 403, and is composed of, for example, a hard disk drive (HDD). The storage 405 stores the control program for the cloud server 103, which is executed by the controller unit 400. Note that the storage 405 is not limited to a hard disk drive, but may be other storage devices with equivalent functionality to a hard disk drive, such as a solid-state drive (SSD).

[0032] The CPU 402 executes a startup program stored in the ROM 404 when the cloud server 103 is powered on or otherwise started up. This startup program reads the control program stored in the storage 405 and loads it onto the RAM 403. By executing the startup program, the CPU 402 executes the control program loaded onto the RAM 403 to control the operation of the entire controller unit 400. The CPU 402 also stores and reads / writes data used when executing the control program onto the RAM 403. The storage 405 can also store various settings necessary when executing the control program. The CPU 402 can communicate with other devices on the network 104 via the network I / F 406.

[0033] <Hardware configuration of the imaging device> Figure 5 is a block diagram showing the hardware configuration of the imaging device 106. As shown in Figure 5, the imaging device 106 has a controller unit 500. The controller unit 500 includes a CPU 502, RAM 503, ROM 504, storage 505, network I / F 506, camera controller 507, and camera 508, which are connected to each other via a system bus 501 so that they can communicate with one another.

[0034] The CPU 502 is a central processing unit that controls the operation of the entire controller unit 500. The RAM 503 is volatile memory. The ROM 504 is non-volatile memory and stores the startup program for the CPU 502. The storage 505 is a storage device with a larger capacity than the RAM 503 and is composed of, for example, a hard disk drive (HDD). The storage 505 stores the control program for the imaging device 106, which is executed by the controller unit 500. Note that the storage 505 is not limited to a hard disk drive, and may be other storage devices with functions equivalent to a hard disk drive, such as a solid-state drive (SSD).

[0035] When the imaging device 106 is powered on or otherwise started up, the CPU 502 executes a startup program stored in the ROM 504. This startup program reads the control program stored in the storage 505 and loads it onto the RAM 503. By executing the startup program, the CPU 502 executes the control program loaded onto the RAM 503 to control the operation of the entire controller unit 500. The CPU 502 also stores and reads / writes data used when executing the control program onto the RAM 503. The storage 505 can also store various settings necessary when executing the control program. The CPU 502 can communicate with other devices on the network 104 via the network I / F 506.

[0036] The camera controller 507 is connected to the camera 508 and controls various exposure and image quality adjustments for shooting. The camera 508 is equipped with a shooting lens and, according to the control of the camera controller 507, captures images of the area around the image forming apparatus 101, i.e., the area to be monitored.

[0037] <Software configuration of image forming apparatus> Figure 6 is a block diagram showing the software configuration of the image forming apparatus 101. As mentioned above, the device control program 5000 shown in Figure 6 is stored in the ROM 204 and is loaded onto the RAM 203 and executed by the CPU 202 when the image forming apparatus 101 is started. The device control program 5000 includes a data transmission / reception unit 5001, a data analysis unit 5002, a job control unit 5003, a data management unit 5004, a display unit 5005, an operation target determination unit 5006, a scan unit 5007, and a print unit 5008.

[0038] The data transmission / reception unit 5001 transmits and receives data via TCP / IP to other devices on the network 104 through the network interface 206. In this embodiment, when a malfunction occurs in the image forming apparatus 101, malfunction information as device information and the status of the recovery operation are transmitted to the cloud server 103.

[0039] The data analysis unit 5002 converts the device operation data received by the data transmission / reception unit 5001 into commands that each module in the device control program 5000 communicates. The data analysis unit 5002 then transmits the corresponding commands to the job control unit 5003, the data management unit 5004, and the display unit 5005.

[0040] The job control unit 5003 controls the print engine 211 and scanner 213 via the print controller 210 and scan controller 212. For example, if the display unit 5005 detects that the start key has been pressed while the copy function screen is displayed, the job control unit 5003 receives the copy job parameters and a job start instruction from the operation target determination unit 5006. Based on these job parameters, the job control unit 5003 controls the print engine 211 to print the image data read by the scanner 213 onto the sheet. The mechanisms for scanning and printing control are publicly known, so no further explanation is provided.

[0041] The data management unit 5004 stores and manages various data, such as work data generated by the execution of the device control program 5000, setting parameters necessary for controlling each device, and user operation history, in designated areas on the RAM 203 and storage 205. The data management unit 5004 also stores and manages authentication information necessary for communication with the gateway 105 and device information necessary for communication with the cloud server 103.

[0042] The display unit 5005 controls the operation panel 209 via the display controller 207. For example, the display unit 5005 displays user-operable UI components (buttons, pull-down lists, checkboxes, etc.) on the operation panel 209. The screen displayed on the operation panel 209 is updated based on screen display control information. The operation target determination unit 5006 obtains the touched coordinates on the operation panel 209 via the operation interface 208 and determines that the user-operable UI component displayed on the operation panel 209 at that time is the operation target.

[0043] The scanning unit 5007 performs a scan using the scanner 213 via the scan controller 212 based on the scan job parameter settings in the job control unit 5003, and saves the scanned image data to the data management unit 5004. The printing unit 5008 performs printing using the print engine 211 via the print controller 210 based on the print job parameter settings in the job control unit 5003.

[0044] <Software configuration of the voice control device> Figure 7 is a block diagram showing the software configuration of the voice control device 102. The voice control program 600 shown in Figure 7 is stored in the storage 305 as described above, and when the voice control device 102 is started, the CPU 302 loads it onto the RAM 303 and executes it. The voice control program 600 includes a data transmission / reception unit 601, a data management unit 602, a voice control unit 603, a voice acquisition unit 604, a voice playback unit 605, a display unit 606, a voice operation start detection unit 607, and a speech end determination unit 608.

[0045] The data transmission / reception unit 601 transmits and receives data via TCP / IP to and from other devices on the network 104 via the network interface 306. The data transmission / reception unit 601 transmits the voice data spoken by the user, acquired by the voice acquisition unit 604, to the cloud server 103. The data transmission / reception unit 601 also receives speech synthesis data used to notify the user of maintenance information. This speech synthesis data is generated on the cloud server 103. The data management unit 602 stores and manages various data, such as work data generated by the execution of the voice control program 600, in a predetermined area on the storage 305. For example, the data management unit 602 stores and manages volume setting data for the voice playback unit 605 and authentication information necessary for communication with the gateway 105. The data management unit 602 also stores and manages device information necessary for communication with the image forming apparatus 101 and the cloud server 103.

[0046] The voice acquisition unit 604 converts the analog voice of a user near the voice control device 102, acquired by the microphone 308, into voice data and temporarily stores it. The user's voice is converted into a predetermined format, such as MP3. This user's voice is then temporarily stored on RAM 303 as encoded voice data for transmission to the cloud server 103. The timing of the start and end of processing by the voice acquisition unit 604 is managed by the voice control unit 603. The encoding of the voice data may be in a general-purpose streaming format, or the encoded voice data may be sequentially transmitted by the data transmission / reception unit 601. The voice playback unit 605 plays back the voice synthesis data (voice message) received by the data transmission / reception unit 601 through the speaker 310 via the audio controller 309. The timing of voice playback by the voice playback unit 605 is managed by the voice control unit 603.

[0047] The display unit 606 displays information on the LED 312 via the display controller 311. For example, the display unit 606 controls the lighting of the LED 312 when the voice operation start detection unit 607 detects that a voice operation has been performed. The timing of the display on the display unit 606 is managed by the voice control unit 603. The voice operation start detection unit 607 detects the wake word spoken by the user or the pressing of the operation start key (not shown) on the voice control device 102, and transmits this detection result to the voice control unit 603 as an operation start notification. The "wake word" is a predetermined voice word. The voice operation start detection unit 607 constantly detects the wake word from the analog voice of the user near the voice control device 102, which is acquired by the microphone 308. The user can operate the image forming apparatus 101 by speaking the wake word, followed by what they want to do.

[0048] The speech termination determination unit 608 determines the timing of the end of processing in the voice acquisition unit 604. For example, the speech termination determination unit 608 determines that the user's speech has ended when the user's voice is interrupted for a predetermined time (e.g., 3 seconds), and transmits the determination result as a speech termination notification to the voice control unit 603. Note that the determination of speech termination is not limited to being based on time without speech (blank time), but may also be based on a predetermined phrase used by the user, for example.

[0049] The voice control unit 603 is the central control unit in the voice control program 600, and controls the other modules within the voice control program 600 to work in coordination with each other. For example, the voice control unit 603 controls the start and end of processing for the voice acquisition unit 604, the voice playback unit 605, and the display unit 606. In addition, after voice data is acquired by the voice acquisition unit 604, the voice control unit 603 controls the transmission of the voice data from the data transmission / reception unit 601 to the cloud server 103. Furthermore, after receiving the synthesized voice data from the cloud server 103 via the data transmission / reception unit 601, the voice control unit 603 controls the playback of the synthesized voice data by the voice playback unit 605.

[0050] <Cloud Server Software Configuration> Figure 8 is a block diagram showing the software configuration of the cloud server 103. The system control program 700 shown in Figure 8 is stored in the storage 405 as described above, and is loaded onto the RAM 403 and executed by the CPU 402 when the cloud server 103 starts up. The system control program 700 includes a data transmission / reception unit 701, a data management unit 702, an image analysis unit 703, a maintenance management unit 704, a user information management unit 708, and a voice data conversion unit 710. The voice data conversion unit 710 includes a voice recognition unit 705, a morphological analysis unit 706, and a voice synthesis unit 707.

[0051] The data transmission / reception unit 701 transmits and receives data via TCP / IP to other devices on the network 104 through the network interface 406. The data transmission / reception unit 701 receives user voice data from the voice control device 102, captured image data from the imaging device 106, and device information data from the image forming apparatus 101. The data management unit 702 holds and manages various data such as voice data generated by the speech synthesis unit 707, analysis models and parameters necessary for performing speech recognition processing in the morphological analysis unit 706, and image feature extraction processing in the image analysis unit 703.

[0052] The image analysis unit 703 analyzes the captured images received from the imaging device 106 to detect people. The image analysis unit 703 also refers to user information held in the user information management unit 708 to perform analysis processing to identify individual users, such as which user it is. The maintenance management unit 704 stores and manages malfunction information and usage status from device information transmitted from the image forming apparatus 101. The maintenance management unit 704 also extracts maintenance information corresponding to the malfunction information.

[0053] As mentioned above, the speech data conversion unit 710 includes a speech recognition unit 705, a morphological analysis unit 706, and a speech synthesis unit 707. The speech recognition unit 705 performs speech recognition processing to convert the user's speech data received by the data transmission / reception unit 701 into text. The speech recognition processing converts the user's speech data into phonemes using an acoustic model, and then converts the phonemes into actual text data using a language model.

[0054] The morphological analysis unit 706 performs morphological analysis on the speech recognition data converted by the speech recognition unit 705, according to the language settings. Morphological analysis derives a sequence of morphemes from a dictionary containing information such as the grammar and parts of speech of the language, and further determines the part of speech of each morpheme. The speech synthesis unit 707 converts the text information contained in the text data into speech information based on the language settings. The morphological analysis unit 706 and the speech synthesis unit 707 are publicly known.

[0055] The user information management unit 708 manages information related to users who use the notification system 100. The user information management unit 708 manages various device settings of the image forming apparatus 101. The system sets and maintains a user-specific level of permission to modify this information. The user information management unit 708 also manages specific symbolic pattern information, such as faces, armbands, and nameplates, to identify individual users. This information is referenced in the user identification process by the image analysis unit 703.

[0056] In this embodiment, specific symbol pattern information for identifying individual users is pre-registered and statically stored as a predetermined pattern model, but is not limited to this. For example, an image analysis process is performed on an image captured by the camera control program 800 in step S2003 (see Figure 15), which will be described later, and changes in feature quantities are detected during this process. These changes in feature quantities can then be appropriately reflected and updated as specific symbol pattern information using machine learning to dynamically respond to changes in the user's appearance. In this case, it is also possible to appropriately perform a user feature update process that identifies and updates changes in feature quantities from a user image captured when the user logs into the image forming apparatus 101.

[0057] <Software configuration of the imaging device> Figure 9 is a block diagram showing the software configuration of the imaging device 106. The camera control program 800 shown in Figure 9 is executed on the CPU 502. This camera control program 800 includes a data transmission / reception unit 801, an imaging unit 802, and a monitoring control unit 803. The data transmission / reception unit 801 has the function of receiving monitoring start and end instructions from the cloud server 103, and the function of converting the data of images captured by the imaging device 106 and transmitting the converted image data to the cloud server 103. The imaging unit 802 has the function of controlling the mounted imaging lens to photograph the target and the function of acquiring the imaging signal. The monitoring control unit 803 has the function of comprehensively interpreting instruction data received from the cloud server 103, controlling the start and end of monitoring processing, and controlling the transmission instructions for captured data.

[0058] <Recovery Procedure Management Table> Figure 10 shows a recovery procedure management table. The recovery procedure management table shown in Figure 10 is used when a malfunction occurs in the image forming apparatus 101 and the malfunction is resolved and the apparatus is restored to working order. It is managed (stored) by the maintenance management unit 704 of the cloud server 103. In this embodiment, the recovery procedure for resolving each type of malfunction that occurs in the image forming apparatus 101 is managed in the recovery procedure management table.

[0059] As shown in Figure 10, the recovery procedure management table stores information on the nature of the malfunction, the recovery procedure, and the role of the user who can perform the recovery. This recovery procedure management table makes it possible to understand who should do what in response to a malfunction. For example, if the malfunction is "ITB belt deterioration," the recovery procedure "ITB belt replacement" is set to be performed by the user who can perform it, "service technician." Similarly, if the malfunction is "conveyor roller wear," the recovery procedure "roller replacement" is set to be performed by the user who can perform it, "service technician." If the malfunction is "waste toner full," the recovery procedure "waste toner box replacement" is set to be performed by the user who can perform it, "administrator user (or service technician)."

[0060] In this embodiment, recovery work and the user who should perform the recovery work are determined according to the nature of the malfunction. Furthermore, users who do not fall under the category of users who should perform the recovery work are notified to prohibit the use of the function related to the malfunction or to recommend that the function be degraded (fallback). This helps to suppress operations that could lead to further malfunctions (secondary malfunctions).

[0061] Therefore, the recovery procedure management table is used to distinguish between users who should be notified of the recovery procedure and those who should not. "Functional degradation" refers to a temporary measure taken when a normally used mechanism or other system malfunctions and ceases to function properly, by limiting its functionality and performance to maintain a limited but usable state.

[0062] <User-type message generation table> Figures 11A and 11B show user-type message generation tables managed by the cloud server. The user-type message generation tables shown in Figures 11A and 11B are tables that contain messages for corrective actions (recovery procedures) corresponding to malfunctions in the image forming apparatus 101, and are managed by the maintenance management unit 704 of the cloud server 103. Furthermore, messages regarding these malfunctions are notified by voice via the speaker 310 (notification step). Thus, in this embodiment, the speaker 310 is included in the notification means for notifying messages. Note that message notification is not limited to voice notification; for example, it may also be notification by image, or by both voice and image.

[0063] The notification system 100 detects users in the vicinity of the image forming apparatus 101 when a malfunction occurs in the image forming apparatus 101 and prompts those users to take appropriate action according to the malfunction via an audio message. However, if a user unfamiliar with this procedure is notified, their inability to take appropriate action may lead to secondary malfunctions.

[0064] Therefore, the content of notifications will be determined according to the user's role. General users will not be notified of the recovery procedure, only that the functionality is restricted. On the other hand, administrator users and service personnel who are familiar with performing the recovery procedure will be proactively notified of the recovery procedure to encourage early recovery. Notifications to administrator users and service personnel will be given when they have business with the image forming apparatus 101 and intentionally approach it.

[0065] Furthermore, notifications to administrator users and service personnel are also sent if they happen to pass by the vicinity of the image forming apparatus 101 and enter the imaging area of ​​the imaging device 106. However, for general users passing by the vicinity of the image forming apparatus 101, even if the user is detected, proactive message notifications are suppressed unless they stand directly in front of the image forming apparatus 101 and begin operating it.

[0066] As shown in Figure 11A, the user-type message generation table contains message content that should be notified according to the role of the general user, administrator user, and service technician, depending on the recovery procedure for the malfunction. Hereinafter, notifications to general users will be referred to as "first notifications," notifications to administrator users as "second notifications," and notifications to service technicians as "third notifications." For example, if the recovery procedure is "ITB belt replacement," general users will be notified as "PRINT / COPY function cannot be used" as the first notification. This prevents the PRINT / COPY function, which uses the ITB belt that needs to be replaced, from continuing to be used even though the ITB belt that needs replacing is available.

[0067] Furthermore, if an administrator user has the authority level to restrict the use of the PRINT / COPY function by forcibly limiting its use by general users, it is preferable to encourage the reduction of the function. In this case, the administrator user will be notified as a second notification, "Please reduce to the SEND function only," so that they can reduce to the SEND function, which does not involve this component. Also, as mentioned above, in the case of "ITB belt replacement," it is necessary for the service technician to perform the recovery work itself (see Figure 10). In this case, the service technician will be notified as a third notification, "Please replace the ITB belt."

[0068] If the recovery procedure is "roller replacement," the user who can perform this is the service technician, so the service technician will be notified as a third notification that "Please replace the transport roller." In addition, general users will be notified as a first notification that "PRINT / COPY functions are unavailable," and administrator users will be notified as a second notification that "Please limit yourself to SEND functions only."

[0069] If the recovery procedure is "replace the waste toner box," the only user who can perform this action is the administrator user. Therefore, the administrator user will receive a second notification stating, "Please replace the waste toner box." General users will receive a first notification stating, "The PRINT / COPY function is unavailable." Note that if the recovery procedure is "replace the waste toner box," no work by a service technician is required, so a third notification to the service technician is deemed "unnecessary" and will not be sent as a message. If the service technician can also perform the "replace the waste toner box," the third notification may be the same as the second notification, "Please replace the waste toner box."

[0070] As described above, the notification system 100 can determine the content of a message to be one of three notifications—a first notification, a second notification, or a third notification—depending on whether the user is a general user, an administrator user, or a service technician. The first notification is a notification of the suspension or restriction of use of the image forming apparatus 101 due to a malfunction. The second notification is a notification of work to degrade the functionality of the image forming apparatus 101 in order to temporarily enable its use even under malfunction conditions. The third notification is a notification of recovery work to resolve the malfunction. By determining the type of notification in this way, the notification system 100 can notify individuals who are highly relevant to the message, thereby improving the human interface. In this embodiment, the maintenance management unit 704 functions as a storage means for pre-storing the first notification, the second notification, and the third notification.

[0071] As shown in Figure 11B, the user-type message generation table contains message content for calling the appropriate person to perform the recovery work, according to the recovery procedure for the malfunction. Hereinafter, when a general user is notified in the first notification, the notification to call an administrator user will be referred to as the "first call notification," and when an administrator user is notified in the second notification, the notification to call a service technician will be referred to as the "second call notification." When a malfunction occurs, it is practically difficult to always detect an available user near the image forming apparatus 101. Therefore, in this embodiment, when the available user cannot be detected, a message prompting contact with the available user is sent.

[0072] Generally, only administrator users can contact service technicians, and general users often do not know how to contact them. Therefore, in this embodiment, when general users need to contact a service technician, they are encouraged to contact an administrator user. The administrator user then receives the contact from the general user and forwards the request for assistance to the service technician.

[0073] As shown in Figure 11B, if the recovery procedure is "ITB belt replacement" and a regular user is detected, the regular user will be notified as a first call notification saying "Please call the administrator user." This allows the regular user to call the administrator user and inform them of the "ITB belt replacement." The administrator user can then request the "ITB belt replacement" from a service technician, thus enabling the rapid recovery of the image forming apparatus 101.

[0074] Furthermore, if the recovery procedure is "ITB belt replacement" and an administrator user is detected, the administrator user will be notified as a second call notification saying "Please call a service technician." This allows the administrator user to request "ITB belt replacement" from a service technician, enabling the rapid recovery of the image forming apparatus 101. The same first and second call notifications as for "ITB belt replacement" are also issued when the recovery procedure is "roller replacement."

[0075] Furthermore, if the recovery procedure is "waste toner box replacement" and a regular user is detected, the regular user will be notified as a first call notification with the message "Please call the administrator user." This allows the regular user to call the administrator user and request "waste toner box replacement," thus enabling the rapid recovery of the image forming apparatus 101. Note that if the recovery procedure is "waste toner box replacement" and an administrator user is detected, a second call notification, i.e., a message prompting contact with a service technician, will not be issued. This is because, as mentioned above, the administrator user will handle "waste toner box replacement."

[0076] As described above, the notification system 100 can issue a first call notification to summon an administrator user when it notifies a general user with a first notification, and a second call notification to summon a service technician when it notifies an administrator user with a second notification. This allows the image forming apparatus 101 to be quickly restored and used.

[0077] In this embodiment, the maintenance management unit 704 functions as a storage means for pre-storing the first call notification and the second call notification. When the first notification is issued, the first notification and the first call notification are repeated, and when the second notification is issued, the second notification and the second call notification are repeated. This further facilitates recovery work. The notification system 100 can also issue or stop the first call notification and the second call notification depending on the nature of the malfunction of the image forming apparatus 101. For example, if the malfunction of the image forming apparatus 101 is a paper jam, it can be handled by a general user, so the first call notification and the second call notification can be stopped.

[0078] <Peripheral monitoring process sequence> Figure 12 is a sequence diagram showing the peripheral monitoring and control process performed between the camera control program 800 and the system control program 700. In this embodiment, the imaging device 106 monitors users present in the vicinity of the image forming apparatus 101 according to the instructions of the system control program 700. The imaging device 106 captures images of the area surrounding the image forming apparatus 101 (the area to be monitored) and continuously transmits the captured image data to the system control program 700.

[0079] The system control program 700 (image analysis unit 703) performs image analysis on the captured image data received from the imaging device 106. The system control program 700 also compares the analysis results with user information held by the user information management unit 708 to identify individual users. User information being identified is retained as identified user as long as the user remains within the imaging range of the imaging device 106. In this way, the system control program 700 functions as an acquisition means to acquire attribute information (roles) related to the attributes of people in an image based on the image captured by the imaging device 106 (acquisition process).

[0080] This attribute information indicates whether the user within the monitored area captured by the imaging device 106 is a general user, an administrator user, or a service technician. As mentioned above, the notification system 100 can determine the content of the message to be one of the first notification, second notification, or third notification depending on the attribute information (role), i.e., general user, administrator user, or service technician. In this way, the system control program 700 functions as a decision means for determining the content of the message according to the attribute information (role) (decision step).

[0081] As shown in Figure 12, in step S101, the system control program 700 sends an instruction signal to the camera control program 800 to start monitoring the area around the image forming apparatus 101. In step S102, the camera control program 800 controls the imaging device 106 to capture images of the area around the image forming apparatus 101 and sends the captured image data to the system control program 700 at predetermined intervals.

[0082] In step S103, the system control program 700 forms an image from the captured image data received in step S102 and performs image analysis processing on the image. This image analysis processing uses at least one feature extraction algorithm from among ORB (Oriented FAST and Rotated BRIEF), SURF (Speeded-Up Robust Features), and SIFT (Scale-Invariant Feature Transform). This allows for accurate and rapid extraction of user features from the image.

[0083] In step S104, the system control program 700 extracts patterns corresponding to the person detection model from the features extracted in step S103 and performs person detection analysis to determine if features that appear to belong to a specific user are present in the captured image. In other words, in step S104, attribute information is obtained based on the extraction results in step S103. This makes it possible to determine whether the user in the monitored area captured by the imaging device 106 is a general user, an administrator user, or a service technician.

[0084] In step S105, the system control program 700 compares the person detection analysis results from step S104 with the user information held in the user information management unit 708 to identify an individual user. The user information identified here is retained as an identified user as long as the user remains within the shooting range of the shooting device 106.

[0085] <Sequence from device anomaly detection to voice notification> Figure 13 is a sequence diagram showing the processing from device abnormality detection to voice notification, which is carried out between the device control program 5000, the system control program 700, and the voice control program 600. When the device control program 5000 detects an abnormality due to its own malfunction, it transmits the malfunction information along with device information to the system control program 700. Based on the device information received from the device control program 5000, the system control program 700 determines the type of malfunction in the image forming apparatus 101.

[0086] Subsequently, the system control program 700 refers to the maintenance management unit 704 to extract the recovery procedure corresponding to the type of malfunction and the user who can perform it. The system control program 700 also refers to the user being detected by the surrounding monitoring process (Figure 12) to determine whether a predetermined user is present in the vicinity of the image forming apparatus 101.

[0087] If the system control program 700 determines that a user exists, it generates a notification message for that user and synthesizes it into speech. The synthesized speech data is sent to the speech control program 600. The speech control program 600 plays back the speech data received from the system control program 700. As a result, the user is notified of the speech data, i.e., the message.

[0088] The voice control program 600 waits for a predetermined time for a response from the user after playing back the voice data. If a response is received from the user, the voice control program 600 records the response as voice data and sends it to the system control program 700. The system control program 700 analyzes the user's response using the voice recognition unit 705 and the morphological analysis unit 706.

[0089] Subsequently, the device control program 5000 transmits recovery operation status information to the system control program 700 when a recovery operation is performed for a malfunction in the image forming apparatus 101. The system control program 700 refers to the analysis results of the user's response and the recovery operation status information to determine whether or not a recovery operation has been performed. If the system control program 700 determines that a recovery operation has not been performed, it again refers to the detected user information, generates a message, synthesizes speech, sends voice data to the voice control program 600, and issues a notification instruction.

[0090] As shown in Figure 13, in step S201, the device control program 5000 detects an abnormal state due to its own malfunction. In step S202, the device control program 5000 transmits its own device information, including the malfunction information, to the system control program 700.

[0091] In step S203, the system control program 700 compares the device information received from the device control program 5000 with the maintenance information managed by the maintenance management unit 704 to determine the cause of the malfunction of the image forming apparatus 101. The system control program 700 also refers to the recovery procedure management table (see Figure 10) to extract the recovery procedure for the malfunction of the image forming apparatus 101 and the users who can perform it. In step S204, the system control program 700 refers to user information regarding users detected near the image forming apparatus 101, which is being monitored by the camera control program 800 using the imaging device 106.

[0092] In step S205, the system control program 700 generates a message from the user-type message generation table (see Figure 11A) that is suitable as recovery procedure information for a malfunction occurring in the image forming apparatus 101 for the detected user. In step S206, the system control program 700 converts the message generated in step S205 into audio data in a predetermined data format. In step S207, the system control program 700 transmits the audio data generated in step S206 to the audio control program 600.

[0093] In step S208, the voice control program 600 plays the audio data received from the system control program 700 through the speaker 310. This notifies the detected user of the message content. In step S209, the voice control program 600 performs control to receive a user response after the message content has been notified. The voice control program 600 waits for a user response to the notification message for a predetermined time, and if a response is received from the user, it records the response as audio data and converts it into a predetermined audio data format. On the other hand, if there is no response from the user within the predetermined time, the voice control program 600 generates empty data meaning "no response". In step S210, the voice control program 600 sends to the system control program 700 either the audio data generated as a user response indicating the response result, or the empty data meaning "no response".

[0094] Furthermore, after transmitting the audio data in step S210, in step S211, if the device control program 5000 detects a recovery operation to resolve the malfunction detected in step S201, it notifies the system control program 700 of this fact. Depending on whether or not this notification is received, the system control program 700 switches whether to repeat the sequence from step S204 again. If the device control program 5000 does not receive notification of a recovery operation within a predetermined period after receiving the response in step S210, it restarts the sequence from step S204. The device control program 5000 then makes an audio notification to the user detected within the monitored area.

[0095] <Flowchart for peripheral monitoring process> Figure 14 is a flowchart showing the surrounding area monitoring process by the camera control program 800. The camera control program 800, in accordance with the monitoring instructions from the system control program 700, photographs the area to be monitored, which is the area around the image forming apparatus 101, and transmits the captured images to the system control program 700 at predetermined intervals. The area to be monitored is not particularly limited and can be, for example, an area with a radius of 2 to 3 mm centered on the image forming apparatus 101. Also, for example, if the image forming apparatus 101 is arranged in a small room, the entire inside of the small room can be the area to be monitored. Furthermore, the imaging process by the camera control program 800 is performed continuously unless there is a stop instruction from the system control program 700, but when a stop instruction is received, imaging stops immediately and the process ends.

[0096] As shown in Figure 14, in step S1001, the CPU 502 receives a command from the system control program 700 to start monitoring the area to be monitored. In step S1002, the CPU 502 starts taking images of the area to be monitored and transfers the captured images to the system control program 700. In step S1003, the CPU 502 determines whether it has received a command to stop from the system control program 700. If the CPU 502 has received a command to stop, the process proceeds to step S1004. On the other hand, if the CPU 502 has not received a command to stop, the process returns to step S1001 and executes the subsequent steps sequentially. In step S1004, the CPU 502 stops taking images of the area to be monitored and terminates the process.

[0097] <Flowchart for detecting and identifying users> Figure 15 is a flowchart showing the process of identifying detected users by the system control program 700. The system control program 700 analyzes the captured images received from the camera control program 800 to detect people, identify detected users, and register the identified user information. The system control program 700 repeats the process of identifying detected users unless the notification system 100 is stopped by the system administrator user. If no people are detected, the registered detected user information is cleared. If the system administrator user issues a stop command, the system control program 700 immediately stops the notification system 100.

[0098] As shown in Figure 15, in step S2001, the CPU 402 sends a command to the camera control program 800 to start monitoring the area to be monitored. In step S2002, the CPU 402 receives captured image data from the camera control program 800. In step S2003, the CPU 402 performs image analysis processing on the captured image data received in step S2002. At least one of the aforementioned ORB, SURF, and SIFT is used for this image analysis processing. This makes it possible to extract user features from the image.

[0099] In step S2004, CPU 402 determines whether a feature pattern exists from the features extracted in step S2003 that matches the detection of a person. If the determination in step S2004 shows that a feature pattern exists, i.e., a person has been detected, the process proceeds to step S2005. On the other hand, if the determination in step S2004 shows that no feature pattern exists, i.e., no person has been detected, the process proceeds to step S2008.

[0100] In step S2005, CPU 402 refers to user information managed by user information management unit 708. In step S2006, CPU 402 compares the features of the person detected in step S2004 with the user information referred to in step S2005, analyzes the features, and identifies the individual user of the person detected in step S2004. If the individual user cannot be identified, the process proceeds with "No identified user". In step S2007, CPU 402 registers and retains the user information identified in step S2006 as "Identified Detected User". This user information is retained as long as the user is detected in the monitored area.

[0101] In step S2008, following step S2007, CPU 402 determines that "no users have been detected in the monitored area" and deletes the user information of the "identified detected users" registered in step S2007. If no "identified detected users" exist at the start of this step, the process proceeds without performing any specific actions.

[0102] In step S2009, CPU 402 determines whether the system administrator user has issued a stop command for the notification system 100. If, as a result of the determination in step S2009, CPU 402 determines that there has been a stop command from the system administrator user, the process proceeds to step S2010. On the other hand, if, as a result of the determination in step S2009, CPU 402 determines that there has been no stop command from the system administrator user, the process returns to step S2002 and the subsequent steps are executed sequentially.

[0103] <Flowchart for Device Anomaly Information Notification Processing> Figure 16 is a flowchart showing the device abnormality information notification process by the device control program 5000. After the image forming apparatus 101 is started, the device control program 5000 constantly monitors its own status. Thereafter, unless the user issues a stop command, the device control program 5000 continues to monitor its status. However, if it detects any abnormal status caused by a malfunction, it sends its own device information along with the malfunction information to the system control program 700.

[0104] Furthermore, the device control program 5000 continues to monitor the status even after transmitting malfunction information and device information, unless a stop command is received from the user. However, if a stop command is received from the user, it performs a shutdown process and immediately terminates the process. Although not shown in the diagram, the notification system 100, after detecting a status abnormality and receiving a recovery operation from a user who is capable of executing the recovery procedure, transmits operation information to the system control program 700, as shown in step S211. Subsequently, recovery control according to the recovery procedure is carried out according to a predetermined recovery flow.

[0105] As shown in Figure 16, in step S3001, the CPU 202 activates each device constituting the image forming apparatus 101 (for example, a transport mechanism having an ITB belt or rollers). In step S3002, the CPU 202 checks the current state of each device in the image forming apparatus 101. In step S3003, the CPU 202 determines whether or not a malfunction has occurred in the device checked in step S3002. If the CPU 202 determines that a malfunction has occurred as a result of the determination in step S3003, the process proceeds to step S3004. On the other hand, if the CPU 202 determines that no malfunction has occurred as a result of the determination in step S3003, the process proceeds to step S3005.

[0106] In step S3004, the CPU 202 transfers device information, along with information about the abnormal state caused by the malfunction, to the system control program 700. In step S3005, the CPU 202 determines whether or not a device stop command has been issued by the user, such as by turning off the power. If the CPU 202 determines in step S3005 that a device stop command has been issued, the process terminates. On the other hand, if the CPU 202 determines in step S3005 that there is no device stop command, the process returns to step S3002 and the subsequent steps are executed sequentially.

[0107] <Flowchart for message generation and speech synthesis transfer processing for detected users> Figure 17 is a flowchart showing the message generation and speech synthesis transfer process for detected users by the system control program 700. When the system control program 700 receives device information including malfunction information from the device control program 5000, it refers to the maintenance management unit 704 and extracts the recovery procedure corresponding to the malfunction. Subsequently, the system control program 700 extracts the users who can perform that recovery procedure.

[0108] Furthermore, the system control program 700 refers to the user information being detected by the camera control program 800's surrounding monitoring process, and if a predetermined user is present in the monitored area, it generates a notification message corresponding to that user and synthesizes it into speech. The synthesized speech data is sent to the speech control program 600. This process is initiated when device information is transmitted from the device control program 5000.

[0109] As shown in Figure 17, in step S4001, the CPU 402 receives device information from the device control program 5000. In step S4002, the CPU 402 determines the device status from the device information received in step S4001 and identifies the cause of the malfunction occurring in the image forming apparatus 101. In step S4003, the CPU 402 refers to the user information of the "identified detected user" detected in the monitored area of ​​the camera control program 800.

[0110] In step S4004, CPU 402 determines whether the "identified detected user" referenced in step S4003 exists within the monitored area. If, as a result of the determination in step S4004, CPU 402 determines that the "identified detected user" exists within the monitored area, the process proceeds to step S4005. On the other hand, if, as a result of the determination in step S4004, CPU 402 determines that the "identified detected user" does not exist within the monitored area, the process terminates.

[0111] In step S4005, the CPU 402 performs message generation processing related to malfunction information for the "identified detected user". The content of the message generated in step S4005 is determined according to the user type of the "identified detected user", which will be described later with reference to Figure 18. In step S4006, the CPU 402 synthesizes and generates audio data corresponding to the message content generated in step S4005 in a predetermined audio data format. In step S4007, the CPU 402 transfers the audio data generated in step S4006 to the audio control program 600, and the process ends.

[0112] <Detailed flowchart of the message generation process> Figure 18 is a flowchart of the message generation process by the system control program 700. Figure 18 is a flowchart of the subroutine in step S4005 of Figure 17. As mentioned above, the notification system 100 distinguishes between general users, administrator users, and service personnel when a malfunction occurs in the image forming apparatus 101, and generates and notifies each user accordingly. The system control program 700 extracts the recovery procedure from the malfunction information received from the device control program 5000 and starts this process after determining that the detected user is near the image forming apparatus 101.

[0113] The system control program 700 compares the malfunction information, the detected user information, and the user type message generation table (see Figures 11A and 11B). This allows the system control program 700 to select and generate a notification message corresponding to the detected user type from the user type message generation table (see Figure 11A). If the message notification is not the first notification and has already been sent to the specified detected user, the process from the message generation process (step S5002) onward is skipped; in other words, message generation is stopped.

[0114] Furthermore, if a certain amount of time has not elapsed since the previous message notification, the processes from the message generation process (step S5002) onward are skipped. In this case, no voice data is generated in the speech synthesis process of step S4006 (see Figure 17), and step S4007 is also skipped. As a result, voice notifications from the voice control program 600 are suppressed. This prevents excessive notifications from being sent each time a user is detected in the monitored area, even though a message has already been sent to the designated user. It also allows for observation to ensure that recovery work is carried out within a predetermined time.

[0115] As shown in Figure 18, in step S5001, the CPU 402 determines whether or not a message notification regarding the malfunction to be detected has never been sent to the designated user. In addition to this determination, in step S5001, the CPU 402 also determines whether or not a recovery operation has been performed on the image forming apparatus 101 even though a predetermined time has elapsed, even if a message has been sent. If, as a result of the determination in step S5001, the CPU 402 determines that this is the first message notification and no message has ever been sent before, or that a recovery operation has not been performed even after a certain period of time has elapsed, the process proceeds to step S5002. On the other hand, if the result of the determination in step S5001 is anything other than the above, the process ends.

[0116] In step S5002, CPU 402 refers to the user type message generation table (see Figure 11A) and generates a message corresponding to the user type of the "identified detected user" for the malfunction. In step S5003, CPU 402 refers to the user type message generation table (see Figure 11B) and determines whether additional recovery work by another user is required for the "identified detected user" user type. If, as a result of the determination in step S5003, CPU 402 determines that additional recovery work by another user is required, i.e., that contact with another user is requested, the process proceeds to step S5004. If, as a result of the determination in step S5003, CPU 402 determines that additional recovery work by another user is not required, the process ends. In step S5004, CPU 402 adds a message prompting contact with another user from the user type message generation table (see Figure 11B), and the process ends.

[0117] <Flowchart for voice notification processing> Figure 19 is a flowchart illustrating the voice notification processing by the voice control program 600. When the voice control program 600 receives voice data from the system control program 700, it plays the voice data from the speaker 310. After playing the voice data, a period of time is set for waiting for a response from the user. Furthermore, when the voice control program 600 receives a response from the user, or when a certain period of no response has elapsed, it converts the response data into data and sends it to the system control program 700. If there is no response, empty data is sent to the system control program 700. In this case, it is determined that there was "no response".

[0118] As shown in Figure 19, in step S6001, the CPU 302 receives audio data from the system control program 700. In step S6002, the CPU 302 plays the audio data received in step S6001 through the speaker 310. This notifies the target user of the message content. In step S6003, the CPU 302 receives a response from the user regarding the message notified in step S6001. This reception involves waiting for a response from the user for a predetermined period and recording the user's voice via the microphone 308. In step S6004, the CPU 302 determines whether a response has been received from the user or whether a certain amount of time has elapsed.

[0119] If the CPU 302 determines, based on the judgment in step S6004, that there has been a response or that a certain amount of time has elapsed, the process proceeds to step S6005. On the other hand, if the judgment in step S6004 is otherwise, the process returns to step S6003 and the subsequent steps are executed sequentially. In step S6005, the CPU 302 converts the audio data recorded in step S6003 into a predetermined audio data format and transfers it to the system control program 700. If there is no response input in step S6004 and a timeout occurs after a certain amount of time has elapsed, empty audio data is generated and transmitted as "no response" information.

[0120] <Overhead view showing the arrangement of the image forming apparatus and imaging apparatus, and the person detection method> Figure 20A is an overhead view showing an example of the arrangement of the image forming apparatus 101, the imaging device 106, and the user to be monitored. Figure 20B is a diagram for explaining facial recognition. Figures 20C to 20F are overhead views showing example arrangements of the image forming apparatus 101, the imaging device 106, and the user to be monitored, respectively. In the notification system 100 of this embodiment, when a malfunction occurs in the image forming apparatus 101, it detects users within the monitored area and notifies them of a message according to the user type (role). General users are not notified of the procedure for resolving the malfunction.

[0121] However, it is preferable that the notification system 100 notifies a general user that a malfunction has occurred and its functions are limited when the user attempts to perform an operation on the image forming apparatus 101, thereby restricting the operation. For this reason, it is preferable not to proactively notify a passerby user, such as user A in Figure 20A, who is not moving in a straight line to the installation location of the image forming apparatus 101, even if the user is detected. On the other hand, it is preferable to proactively notify a user, such as user B in Figure 20A, who is moving in a straight line to the installation location of the image forming apparatus 101, in order to suppress unnecessary operations.

[0122] Whether or not a user is moving linearly toward the image forming apparatus 101 can be determined, depending on the installation position of the imaging device 106, by means of, for example, the following embodiments. As shown in Figure 20A, if camera A is installed in approximately the same position and orientation as the image forming apparatus 101, a user approaching so as to face the front of camera A can be determined to be moving linearly toward the image forming apparatus 101. This determination is made possible by facial recognition. As shown in Figure 20B, facial recognition determines whether the user is facing forward, to the left, or to the right of camera A based on the relative positions of the eyes, nose, and mouth extracted as feature points for facial recognition. If it is determined that the user is facing forward toward camera A and has been approaching for a certain period of time, it is determined that the target user is moving linearly toward the image forming apparatus 101, and a message is sent.

[0123] On the other hand, if camera B is installed in a different location from the image forming apparatus 101, the system determines whether the user is moving in a straight line based on the relationship between the installation location of the image forming apparatus 101, the user's location, and the user's direction of movement. User A is not determined to be moving in a straight line because they are moving in a direction different from the straight line connecting their location and the installation location of the image forming apparatus 101. User B is determined to be moving in a straight line because they are moving in the same direction as the straight line connecting their location and the installation location of the image forming apparatus 101.

[0124] Furthermore, the notification system 100 can also determine whether or not to notify a message based on the user's direction of movement. That is, the notification system 100 can also detect whether the user is approaching the image forming apparatus 101 or simply passing by, and determine whether or not to notify a message. This determination is made, for example, by the controller unit 400 (CPU 402) of the cloud server 103. If the user detected within the monitored area is either an administrator user or a service technician, the system proactively notifies a message even if the user is just passing by, prompting them to degrade functionality or replace parts in response to a malfunction.

[0125] Figure 20C shows a situation where an administrator user is proactively notified with a message prompting them to perform a degraded setting (degraded operation) when detected as a passerby user. Figure 20D shows a situation where a service technician is proactively notified with a message prompting them to replace parts when detected as a passerby user.

[0126] On the other hand, for general users passing by, proactive message notifications are suppressed because unnecessarily notifying them of parts replacement or functional degradation could cause malfunctions due to unnecessary operations. For such general users, it is preferable to notify them of restricted functions via message only when they approach the image forming apparatus 101 with the intention of using it. Figure 20E shows a situation where no message notification is sent when a general user passes by the image forming apparatus 101, but a message notification about restricted functions is sent when a general user approaches the image forming apparatus 101.

[0127] Furthermore, as a user type, guest users who are not internal users and cannot be individually identified do not have the authority to operate the image forming apparatus 101. In this case, it is preferable not to notify guest users of messages not only when they pass by, but also when they approach the image forming apparatus 101. Figure 20F shows a situation in which proactive message notification is suppressed when a guest user passes by the image forming apparatus 101 or approaches the image forming apparatus 101.

[0128] <Examples of characteristics used as criteria for determining user type> Figure 21 shows an example of the characteristics used to determine user type from images captured by the imaging device 106. The notification system 100 notifies an unspecified number of users detected within the monitored area with messages regarding malfunctions, but the content of these messages differs depending on the user type. In this configuration, instead of using facial recognition to determine user type, the system detects the clothing and accessories worn by the user and uses the detection results.

[0129] As shown in Figure 21, clothing is detected from areas other than the user's face. If the detected clothing includes a tie or a designated employee ID, the user is identified as a general user. If the clothing is a suit, the user is identified as an administrator user, and if the clothing is a designated work uniform, the user is identified as a service technician. If the clothing is anything other than these, the user is identified as a guest user. Note that the user type determination configuration is not limited to the above configuration. For example, in an environment with a user count management system that manages user types on an individual user basis, it is also possible to identify an individual using facial recognition, extract the user type of that individual user, and then determine the message content according to the extraction result.

[0130] <Detailed flowchart of user-type message generation process> Figure 22 is a flowchart of the user-type message generation process. Figure 22 is a flowchart of the subroutine in step S5002 of Figure 18. As mentioned above, the notification system 100 controls message notifications based on the user-type message generation table (see Figures 11A and 11B), determining the message content to be notified according to the user type.

[0131] Furthermore, the notification system 100 also detects whether the user is approaching the image forming apparatus 101 or simply passing by, and determines whether or not to notify the user of a message. If the user detected within the monitored area is a service technician or administrator, the system proactively notifies the user of a message even if they are just passing by.

[0132] Furthermore, if a user detected within the monitored area is a regular user or a guest user, no message notification will be sent even if the user is just passing through. If a regular user attempts to use a function related to a malfunction, potentially causing further malfunctions, a message will be sent only when the user approaches the image forming apparatus 101, informing them of the restricted function.

[0133] As shown in Figure 22, in step S5101, the CPU 402 determines whether the clothing of the detected user matches the characteristics of a service technician's work uniform. If the CPU 402 determines that the clothing matches the characteristics of a service technician's work uniform, the process proceeds to step S5102. On the other hand, if the CPU 402 determines that the clothing does not match the characteristics of a service technician's work uniform, the process proceeds to step S5103. In step S5102, the CPU 402 determines that the detected user is a service technician, generates a message from the user type message generation table notifying the user of a parts replacement request, and transitions to a control mode for sending this message.

[0134] In step S5103, the CPU 402 determines whether the user to be detected can be identified as an individual user. If the CPU 402 determines that the user to be detected can be identified as an individual user, the process proceeds to step S5103. On the other hand, if the CPU 402 determines that the user to be detected cannot be identified as an individual user, the user to be detected is considered a guest user, and the process proceeds to step S5104. In step S5104, the CPU 402 determines that the user to be detected is a guest user, and transitions to a control mode in which no messages are generated and no message notifications are sent. Note that step S5104 can be omitted as a control step.

[0135] In step S5105, the CPU 402 individually identifies the user to be detected and determines whether or not this user is an administrator user. If the CPU 402 determines that the user is an administrator user as a result of the determination in step S5105, the process proceeds to step S5106. On the other hand, if the CPU 402 determines that the user is not an administrator user as a result of the determination in step S5105, the process proceeds to step S5107. In step S5106, the CPU 402 generates a message from the user type message generation table recommending that the functions that cannot be used due to the malfunction be degraded, and then transitions to a control mode to notify the user of this message.

[0136] In step S5107, the CPU 402 individually identifies the user to be detected and determines whether or not this user is a regular user. If the CPU 402 determines that the user is a regular user as a result of the determination in step S5107, the process proceeds to step S5108. On the other hand, if the CPU 402 determines that the user is not a regular user as a result of the determination in step S5107, the detected user is judged as an undefined user that is neither a guest user nor any other user, an error occurs, the generation of a message is omitted, and the process terminates.

[0137] In step S5108, the CPU 402 determines whether an individually identified general user is approaching the image forming apparatus 101. If, as a result of the determination in step S5108, the CPU 402 determines that a general user is approaching, the process proceeds to step S5109.

[0138] On the other hand, if the CPU 402 determines, as a result of the judgment in step S5108, that no general user is approaching, the process proceeds to step S5110. In step S5109, a message notifying of the functional limitations due to the malfunction is generated from the user type message generation table, and the system transitions to a control mode that notifies this message. In step S5110, the CPU 402, similar to step S5104, does not generate any messages and transitions to a control mode that does not notify any messages. Note that step S5110 can also be omitted as a control step.

[0139] As explained above, the notification system 100 uses the imaging device 106 to monitor for the presence of a user approaching the image forming apparatus. If a user is detected, the user's image data is sent to the cloud server 103 for user identification. Furthermore, if a malfunction occurs in the image forming apparatus 101, device information is transferred to the cloud server 103 at the time the abnormality is detected, and the cause and recovery methods are referenced.

[0140] Furthermore, appropriate messages are internally generated and sent to the user according to the identified user type. If additional recovery work is required by another user, information about the recovery work is also included in this notification. In addition, this notification process does not send a notification every time a user is detected. Once a notification has been sent, it waits for a predetermined time to see if recovery work is performed, and only sends another notification if the recovery work is still not performed. In other words, if there are no changes in attribute information, the notification of a message and the suspension of notifications are repeated at predetermined intervals.

[0141] This allows for the notification of malfunction-related messages tailored to the user type while suppressing unnecessary notifications, thereby preventing further malfunctions. It also enables prompting recovery procedures only to the appropriate users. Furthermore, the notification system 100 switches the message content each time attribute information changes. This allows for the notification of messages appropriate to the attribute information.

[0142] In this embodiment, the system detects when a user approaches the image forming apparatus 101 based on the image captured by the imaging device 106, but it is not limited to this. For example, a short-range communication means (such as BLE (Bluetooth Low Energy)) capable of detecting when a user approaches the image forming apparatus 101 may be used. Also, in this embodiment, the cloud server 103 holds user information and performs user determination, but it is not limited to this. For example, in the event of a malfunction, the cloud server 103 may simultaneously transfer user information it holds along with device information transferred from the image forming apparatus 101 to perform user determination.

[0143] Furthermore, the configuration for stopping messages is not particularly limited; for example, an operation button for stopping messages may be provided. Alternatively, messages may be stopped after a predetermined number of messages have been issued. In this case, the number of messages issued may be changed depending on whether the user is a general user, an administrator user, or a service technician. Messages may also be stopped when a user leaves the monitored area. Messages may also be stopped when the recovery operation for the image forming apparatus 101 is completed. If the recovery operation for the image forming apparatus 101 is not performed after a certain period of time, a message notification may be sent again according to the detected user.

[0144] Furthermore, in this embodiment, the notification system 100 is configured to acquire attribute information regarding the attributes of a user (person) in an image based on an image captured by the camera 106, but the system is not limited to this configuration. For example, if an IC tag is incorporated into an employee ID card, the system may receive a signal from the IC tag in a receiving unit (detection means) to detect the user, and then acquire attribute information regarding the user's attributes based on the detection result.

[0145] <Second Embodiment> Next, a notification system according to a second embodiment of the present invention will be described. In the second embodiment, the user is identified by facial recognition, and the content of the message is determined based on the registered user's role (authority and affiliation). In the following description, the configuration and processing common to the first embodiment will be omitted from the description, and only the differences will be described.

[0146] <Overall Structure> The overall hardware configuration of the notification system 100, the image forming apparatus 101, the voice control device 102, the shooting device 106, and the cloud server 103 is the same as that shown in Figures 1 to 5. The cloud server 103 can determine words related to screen operation and job execution of the image forming apparatus 101 from the voice recognition results of the voice control device 102. A "job" is the execution unit of a series of image forming processes (e.g., copying, scanning, printing, etc.) realized by the image forming apparatus 101 using the print engine 211 or the scanner 213. The cloud server 103 also generates a predetermined message (text) according to the voice recognition result or the word determination result, and synthesizes voice data for voice playback of the message content by the voice control device 102.

[0147] In this embodiment, there are two types of individuals: a general user (user) and an administrator user (administrator). Service personnel and guest users may also be included as individuals. Guest users are, in principle, visitors or others who are prohibited from using the image forming apparatus 101.

[0148] The software configuration of the image forming apparatus 101 is the same as that shown in Figure 6. Specifically, the data transmission / reception unit 5001 transmits and receives data via TCP / IP to other devices on the network 104 via the network interface 206. The data transmission / reception unit 5001 receives device operation data generated by the cloud server 103. In this embodiment, the data transmission / reception unit 5001 also sends screen update notifications to the cloud server 103 indicating that the screen display content of the job execution result and device operation result response has been updated, and job execution status notifications indicating the status of the job. The content of these notifications will be described later with reference to the sequence diagram shown in Figure 24.

[0149] The data management unit 5004 stores and manages various data, such as work data generated by the execution of the device control program 5000, setting parameters necessary for each device control, and various user operation histories, in predetermined areas on the RAM 203 and storage 205. Other examples of such data include job data consisting of combinations of setting items and setting values ​​for jobs executed by the job control unit 5003 (described later), and language settings as information on the language displayed on the operation panel 209.

[0150] Furthermore, the data management unit 5004 also stores and manages authentication information necessary for communication with the gateway 105, device information necessary for communication with the cloud server 103, etc. The data management unit 5004 also stores and manages image data formed by the image forming apparatus 101. In addition, the data management unit 5004 stores screen control information used for screen display control in the display unit 5005, and operation target determination information used for determining the operation target in the operation target determination unit 5006.

[0151] The data management unit 5004 manages screen control information and operation target determination information for each screen displayed by the display unit 5005. The data management unit 5004 also stores data that can identify users. Examples of data stored in the data management unit 5004 include user face data and calculated values ​​of the user's facial features. In addition, the image forming apparatus 101 can also determine whether the user is a match or not by comparing image data acquired by the camera 508 with the data in the data management unit 5004, for example, in the data analysis unit 5002.

[0152] The display unit 5005 controls the operation panel 209 via the display controller 207. For example, the display unit 5005 displays user-operable UI components (buttons, pull-down lists, checkboxes, etc.) on the operation panel 209. The screen displayed on the operation panel 209 is updated based on screen display control information. For example, the display unit 5005 retrieves a language dictionary corresponding to the language settings stored in the data management unit 5004 from the storage 205 and displays text data based on the language dictionary on the screen.

[0153] The operation target determination unit 5006 obtains the touched coordinates on the operation panel 209 via the operation I / F 208 and determines that the user-operable UI component displayed on the operation panel 209 at that time is the operation target. The operation target determination unit 5006 also reads the screen display control information corresponding to the UI component determined to be the operation target and determines the processing content when the operation is received based on this information. For example, the operation target determination unit 5006 may issue an instruction to the display unit 5005 to update the screen display content, or transmit the job parameters set by the user operation and the instruction to start the job to the job control unit 5003.

[0154] The software configuration of the voice control device 102 is the same as that shown in Figure 7. However, the data transmission / reception unit 601 receives speech synthesis data that will be used as a response to the user. The voice processing after the voice operation start detection unit 607 detects the wake word will be described later.

[0155] The speech termination determination unit 608 determines the timing of the end of processing in the voice acquisition unit 604. For example, the speech termination determination unit 608 determines that the user's speech has ended when the user's voice is interrupted for a predetermined time (e.g., 3 seconds), and transmits the determination result as a speech termination notification to the voice control unit 603. Note that the determination of speech termination is not limited to being based on time without speech (blank time), but may also be based on predetermined words used by the user. For example, the speech termination determination unit 608 may determine that speech has ended without waiting for a predetermined time if the user uses predetermined words such as "yes," "no," "OK," "cancel," "end," or "start." Furthermore, the determination of speech termination may be performed by the cloud server 103, or it may be based on the meaning and context of the user's speech.

[0156] Here, we will explain the timing of the start and end of processing for the voice acquisition unit 604, the voice playback unit 605, and the display unit 606. The voice control unit 603 starts processing for the voice acquisition unit 604 when it receives an operation start notification from the voice operation start detection unit 607. The voice control unit 603 also terminates processing for the voice acquisition unit 604 when it receives a speech end notification from the speech end determination unit 608.

[0157] For example, suppose a user speaks a wake word, followed by the phrase "I want to copy." At this point, the voice operation start detection unit 607 detects the voice of the wake word and sends an operation start notification to the voice control unit 603. Upon receiving the operation start notification, the voice control unit 603 controls the voice acquisition unit 604 to begin processing.

[0158] The voice acquisition unit 604 acquires the analog voice of the user who then says "I want to copy," converts it into voice data, and temporarily stores it. The speech end determination unit 608 sends a speech end notification to the voice control unit 603 if it determines that there has been a predetermined amount of time of silence after the utterance of "I want to copy." When the voice control unit 603 receives the speech end notification, it terminates the processing of the voice acquisition unit 604. The state from when the voice acquisition unit 604 starts processing until it finishes is referred to as the "speech processing state." The display unit 606 lights up the LED 312 to indicate that it is in the speech processing state.

[0159] After determining that the user has finished speaking, the voice control unit 603 controls the voice acquisition unit 604 to send the temporarily stored voice data to the cloud server 103 via the data transmission / reception unit 601, and waits for a response from the cloud server 103. The response from the cloud server 103 may include, for example, a response message consisting of a header indicating that it is a response and speech synthesis data.

[0160] When the voice control unit 603 receives a response message from the data transmission / reception unit 601, it controls the voice playback unit 605 to play back synthesized speech data. The synthesized speech data is, for example, "Display copy screen." The state from the time the end of speech is determined until the end of playback of the synthesized speech data is referred to as the "response processing state."

[0161] The display unit 606 blinks LED 312 to indicate that it is in response processing mode. After response processing, as long as the dialogue session with the cloud server 103 continues, the user can continue to speak what they want to do without speaking a wake word. The end of the dialogue session is determined when the cloud server 103 sends a dialogue session end notification to the voice control device 102. The state from the end of one dialogue session until the start of the next dialogue session is referred to as the "standby state". The voice control device 102 remains in the standby state until it receives an operation start notification from the voice operation start detection unit 607. The display unit 606 turns off LED 312 during the standby state.

[0162] <Cloud Server Software Configuration> Figure 23 is a block diagram showing the software configuration of the cloud server in the second embodiment. The data conversion control program (system control program) 1700 shown in Figure 23 is stored in storage 405 and is loaded onto RAM 403 and executed by the CPU 402 when the cloud server 103 starts up. The data conversion control program 1700 includes a data transmission / reception unit 701, a data management unit 702, a device operation data generation unit 1703, a user determination unit 1708, and a voice data conversion unit 710. The voice data conversion unit 710 includes a voice recognition unit 705, a morphological analysis unit 706, and a voice synthesis unit 707.

[0163] The data transmission / reception unit 701 transmits and receives data via TCP / IP to other devices on the network 104 through the network interface 406. The data transmission / reception unit 701 receives user voice data from the voice control device 102. The data management unit 702 also transmits the user ID determined by the user determination unit 1708 and the text data determination result generated by the voice recognition processing of the voice recognition unit 705.

[0164] The data management unit 702 stores and manages various data, such as work data generated by the execution of the data conversion control program 1700 and parameters necessary for speech recognition processing by the speech data conversion unit 710, in a predetermined area of ​​the storage 405. For example, the data management unit 702 stores and manages acoustic models and language models for converting speech data received by the data transmission / reception unit 701 into text in a predetermined area of ​​the storage 405.

[0165] Furthermore, the data management unit 702 stores and manages a dictionary for performing morphological analysis of text by the morphological analysis unit 706 in a predetermined area of ​​the storage 405. In addition, the data management unit 702 stores and manages a user ID list for determining the user ID by the user determination unit 1708 in a predetermined area of ​​the storage 405.

[0166] Furthermore, the data management unit 702 stores and manages a voice database for voice synthesis performed by the voice synthesis unit 707 in a predetermined area of ​​the storage 405. The data management unit 702 also stores and manages device information necessary for communication with the voice control device 102 and the image forming apparatus 101. The device operation data generation unit 1703 generates operation data when the image forming apparatus 101 is operated by a user.

[0167] As mentioned above, the speech data conversion unit 710 includes a speech recognition unit 705, a morphological analysis unit 706, and a speech synthesis unit 707. The speech recognition unit 705 performs speech recognition processing to convert the user's speech data received by the data transmission / reception unit 701 into text. The speech recognition processing converts the user's speech data into phonemes using an acoustic model, and then converts the phonemes into actual text data using a language model.

[0168] Note that the input speech may be in multiple languages. In this case, a first speech recognition method may be used that determines the language of the input speech data and converts it into text data corresponding to that language. Alternatively, a second speech recognition method may be used that converts the input speech data into phonemes using acoustic models for different languages, and then converts it into text data for each language using the corresponding language model and outputs it. In the case of the second speech recognition method, since it converts into text data in multiple language formats, the speech recognition unit 705 outputs speech recognition data consisting of text and language settings as a result of the speech recognition process.

[0169] In this embodiment, the input speech languages ​​are Japanese and English. The speech recognition data in Japanese consists of text composed of one or more kana characters and a language setting of "Japanese". The speech recognition data in English consists of text composed of one or more alphabet characters and a language setting of "English". The speech recognition process for converting speech data into speech recognition data is not limited to the above process. The morphological analysis unit 706 performs morphological analysis on the speech recognition data converted by the speech recognition unit 705 according to the language setting.

[0170] Morphological analysis derives a sequence of morphemes from a dictionary containing information such as the grammar and parts of speech of the language, and further determines the part of speech of each morpheme. The morphological analysis unit 706 is implemented using known morphological analysis software such as JUMAN, Chasen, or MeCab. The speech synthesis unit 707 converts the text information contained in the text data into speech information based on the language settings. The morphological analysis unit 706 and the speech synthesis unit 707 are publicly known.

[0171] The user determination unit 1708 determines, based on the user information received by the data transmission / reception unit 701, whether or not the user matches the user ID list registered in the data management unit 702. If the user determination unit 1708 determines that the user matches the user ID list, it also determines whether or not the user has the necessary authority over the image forming apparatus 101.

[0172] The software configuration of the imaging device 106 is the same as that shown in Figure 9. The data transmission / reception unit 801 has the functions of receiving monitoring start and end instructions, converting the data of images captured by the imaging device 106, and transmitting the converted image data.

[0173] Monitoring start and end instruction data are generated by the image forming apparatus 101 or the cloud server 103 (in this embodiment, the image forming apparatus 101). Image data is also transmitted to the image forming apparatus 101 or the cloud server 103 (in this embodiment, the image forming apparatus 101). The imaging unit 802 has the function of controlling the mounted imaging lens to photograph the target object and acquiring the imaging signal. The monitoring control unit 803 has the function of comprehensively interpreting monitoring start and end instruction data, controlling the start and end of monitoring processing, and controlling the transmission instruction of captured data.

[0174] <Sequence of processes performed between the camera control program, device control program, audio control program, and data conversion control program> Figure 24 is a sequence diagram showing the processing performed between the camera control program 800, the device control program 5000, the voice control program 600, and the data conversion control program 1700. In the sequence shown in Figure 24, a user approaching the image forming apparatus 101 within the monitored area is detected and the individual user is identified (discriminated). Then, depending on the identification result, the status of the image forming apparatus 101 is notified by voice or not.

[0175] In the example sequence shown in Figure 24, the image forming apparatus 101 is in a state where the remaining amount of K toner (black toner) is low, and the user should be notified to prepare and replace a new K toner cartridge. The replacement of the K toner cartridge is performed by a user belonging to the first group. The "first group" refers to a group that has predetermined authority over events in the image forming apparatus 101; in this embodiment, it refers to the administrator group authorized to perform the replacement work in order to manage the state of the image forming apparatus 101. In this case, the replacement of the K toner cartridge is performed by an administrator user.

[0176] On the other hand, in addition to the first group, there is also a second group. The "second group" is a group with different privileges than the first group, that is, in this embodiment, a group of users who use the image forming apparatus 101. General users belong to the second group. In the example sequence shown in Figure 24, a user who approaches the image forming apparatus 101 within the monitored area is assumed to be an administrator user belonging to the first group.

[0177] As shown in Figure 24, in step S800, the device control program 5000 instructs the camera control program 800 to begin detecting a user. User detection is initiated, for example, when the image forming apparatus 101 is powered on or when user detection is permitted from the operation panel 209. In step S801, the camera control program 800 begins detecting a user approaching the image forming apparatus 101 within the monitored area. The monitored area is not particularly limited and can be, for example, an area with a radius of 2 to 3 m centered on the image forming apparatus 101. Also, for example, if the image forming apparatus 101 is located in a small room, the entire inside of the small room can be considered the monitored area.

[0178] In step S802, the camera control program 800 captures a detection image with the camera 508 to identify the user. Since user identification is performed by facial recognition, which detects a person's face, the detection image captured in step S802 includes the user's face. In step S803, the camera control program 800 transmits the detection image to the device control program 5000.

[0179] In step S804, the device control program 5000 instructs the data analysis unit 5002 to perform face detection processing on the detection image captured in step S803 to determine whether or not a face is present. If the data analysis unit 5002 determines that a face is present, it calculates the similarity between the detection image and the user's face recognition image stored in the storage 205, and determines that the user with the highest similarity, whose similarity is above a threshold, will be the target for identification. This makes it possible to determine whether the user in the monitored area captured by the imaging device 106 is a general user or an administrator user.

[0180] In this embodiment, the data analysis unit 5002 identifies users in images captured by the imaging device 106. Based on this identification result, the data analysis unit 5002 functions as an acquisition means to obtain the group (first group or second group) to which the user in the monitored area belongs. For example, if a user in the monitored area is identified as an administrator user, it is determined that this administrator user belongs to the first group, i.e., they are associated with it.

[0181] Furthermore, if a user within the monitored area is identified as a general user, it is determined that this general user belongs to the second group. It is preferable that the user identification process performed by the data analysis unit 5002 uses at least one of ORB, SURF, or SIFT. This allows for accurate and rapid extraction of user features from the detected image, thereby enabling accurate user identification.

[0182] In step S805, the device control program 5000 obtains status information of the image forming apparatus 101 held in RAM 203 and the job status received by the image forming apparatus 101. In step S806, the device control program 5000 transmits the user name and personal ID held in the storage 205 of the image forming apparatus 101 as user information identified in step S804 to the data conversion control program 1700.

[0183] In step S807, the device control program 5000 transmits the status information and job status information of the image forming apparatus 101 acquired in step S805 to the data conversion control program 1700 as device information. In step S808, the device control program 5000 transmits message information for a specific user, i.e., for a user belonging to the first group, according to the status information and job status held in the RAM 203 of the image forming apparatus 101.

[0184] In step S809, the data conversion control program 1700 performs processing according to the message information notified from the image forming apparatus 101. Details of this step S809 will be described later with reference to Figure 25.

[0185] In step S810, the voice control program 600 receives the speech synthesis data generated by the data conversion control program 1700 from the data transmission / reception unit 601. In step S811, the voice control program 600 receives a notification that the speech synthesis data generated by the data conversion control program 1700 has finished being processed from the data transmission / reception unit 601.

[0186] In step S812, the voice control program 600 plays back the speech synthesis data received by the voice playback unit 605 in step S810. The speech synthesis data is not particularly limited and could include, for example, "The amount of K toner is low. Please replace it with a new K toner cartridge." The speech synthesis data is generated from the speaker 310. In this way, messages regarding malfunctions of the image forming apparatus 101 are notified by voice via the speaker 310 (notification step).

[0187] In this embodiment, the speaker 310 functions as a notification means for notifying messages. This allows administrator users to proactively recognize messages and prompt them to replace the K toner cartridge. Note that message notification is not limited to voice notifications; for example, it may also be an image notification, or a notification using both voice and image. After the voice synthesis data is played back, the voice control program 600 switches the voice control device 102 to a standby state.

[0188] As described above, it is preferable that the notification system 100 includes in its messages the cause of the malfunction of the image forming apparatus 101 (insufficient K toner) and the appropriate countermeasure for that malfunction (replacement with a new K toner cartridge). This allows the administrator user to accurately understand the status of the image forming apparatus 101. However, the messages are not limited to including both the cause of the malfunction of the image forming apparatus 101 and the appropriate countermeasure for that malfunction; for example, they may include only one of these.

[0189] Furthermore, while the information provided may include, but is not limited to, information on recovery procedures to resolve a malfunction (such as replacing the K toner cartridge), it may also include information on procedures to temporarily degrade the functionality of the image forming apparatus 101, enabling its use even under malfunction conditions. Examples of information on functional degradation procedures include, for instance, if the image forming apparatus 101 is configured for color printing, operations to perform printing using multiple toners other than K toner. This allows printing to be performed using the image forming apparatus 101 temporarily.

[0190] Furthermore, the notification system 100 can notify users who belong to the first group, if the user identified by the data analysis unit 5002 belongs to the first group, and suppress notification of the message if the user identified by the data analysis unit 5002 belongs to the second group. This allows the notification to be sent to users who are highly relevant to the message. In this embodiment, when notifying users of low K toner levels and the need to replace the K toner cartridge, the notification can be sent to administrator users who belong to the first group and are highly relevant to the message.

[0191] Furthermore, the configuration for stopping messages is not particularly limited; for example, an operation button for stopping messages may be provided. In addition, message notifications may be stopped when a predetermined number of messages have been sent. Message notifications may also be stopped when the user leaves the monitored area. Furthermore, message notifications may be stopped when the recovery operation for the image forming apparatus 101 is completed. If the recovery operation for the image forming apparatus 101 is not performed after a certain period of time, the message may be sent again.

[0192] The surrounding area monitoring process by the camera control program is the same as the flowchart shown in Figure 14. The camera control program 800, in accordance with the monitoring instructions from the device control program 5000, photographs the area to be monitored, which is the surrounding area of ​​the image forming apparatus 101, and transmits the captured images to the device control program 5000 at predetermined intervals. Furthermore, the imaging process by the camera control program 800 is performed continuously unless there is a stop instruction from the device control program 5000, but if a stop instruction is received, imaging stops immediately and the process ends.

[0193] <Voice message control processing by data conversion control program> Figure 25 is a flowchart showing the voice message control process up to voice notification by the data conversion control program 1700. Figure 25 is a flowchart showing the detailed processing in step S809 in Figure 24.

[0194] As shown in Figure 25, in step S901, the CPU 402 receives the username and user ID transmitted in step S806 as user information detected by the imaging device 106. This username and other information are stored in the storage 205 of the image forming apparatus 101. In step S902, the CPU 402 receives the status information and job status information of the image forming apparatus 101 transmitted in step S807 as device information.

[0195] In step S903, the CPU 402 receives the message information sent in step S808, depending on the device status information and job status. In step S904, the CPU 402 refers to the user data list held in RAM 403 based on the user ID received in step S901. The CPU 402 then determines whether the user ID exists in the user data list. The user data list will be described later with reference to Figure 26. If the CPU 402 determines, as a result of the determination in step S904, that the user ID exists in the user data list, the process proceeds to step S905. On the other hand, if the CPU 402 determines, as a result of the determination in step S904, that the user ID does not exist in the user data list, the process terminates. This prevents the notification of unnecessary messages.

[0196] In step S905, the CPU 402 determines the type of message information received in step S903. If, as a result of the determination in step S905, the CPU 402 determines that the message is intended for a specific user, i.e., an administrator user belonging to the first group, the process proceeds to step S909. On the other hand, if the CPU 402 determines that the message is not intended for a specific user, the process proceeds to step S906.

[0197] In step S906, the CPU 402 determines whether the type of message information received in step S903 is a message notifying an exceptional state (malfunction) of the image forming apparatus 101. Specifically, the CPU 402 checks whether it is a resolvable device stop request, such as a JAM or no toner remaining in the image forming apparatus 101.

[0198] If the CPU 402 determines, based on the judgment in step S906, that it is a device stop request, the process proceeds to step S907. On the other hand, if the CPU 402 determines, based on the judgment in step S906, that it is not a device stop request, the process proceeds to step S910. In this case, the message is one that can be sent to any user. Therefore, regardless of whether the user identified by the data analysis unit 5002 belongs to the first group or the second group, a message that can be sent to any user is sent. For example, in step S910, the voice message "Copying is complete" is synthesized as a job completion message. This allows any user to recognize that the copy is complete.

[0199] In step S907, the CPU 402 searches the user data list (user information list) from the beginning based on the user ID received in step S901. The CPU 402 then retrieves information regarding user privileges associated with the user ID from the user data list. As mentioned above, there are two types of users who are imaged within the monitored area: general users and administrator users. User privileges are divided into privileges for general users and privileges for administrator users, depending on the content of the exception state of the image forming apparatus 101 that can be resolved.

[0200] In step S908, the CPU 402 executes processing according to the user privileges acquired in step S907. Specifically, the CPU 402 determines whether the user privileges acquired in step S907 are the privileges that can resolve the exceptional state of the image forming apparatus 101, that is, whether the user has the user privileges that can resolve the exceptional state of the image forming apparatus 101. If, as a result of the determination in step S908, the CPU 402 determines that the user has the user privileges, the process proceeds to step S910. On the other hand, if, as a result of the determination in step S908, the CPU 402 determines that the user does not have the user privileges, the process terminates. This prevents the notification of unnecessary messages.

[0201] In step S909, the CPU 402 determines whether the user ID received as user information in step S901 matches the user ID obtained in step S903. If the CPU 402 determines that the user IDs match as a result of the determination in step S909, the process proceeds to step S910. On the other hand, if the CPU 402 determines that the user IDs do not match as a result of the determination in step S909, the process terminates.

[0202] In step S910, the CPU 402 synthesizes voice data to be sent to the voice control device 102. For example, if an administrator user is notified that an exception has occurred, the voice data "An exception has occurred" is synthesized as a message to that effect. Also, if the message type is a received print, the voice data "A print job has arrived for user AAA" is synthesized as a message to that effect. In step S911, the CPU 402 sends the voice synthesis data obtained in step S910 to the voice control device 102. As a result, the voice control device 102 outputs the voice synthesis data as voice.

[0203] <User Data List> Figure 26 shows an example of a user data list. The user data list shown in Figure 26 has items such as User ID, email address, and each user's role, and information for each item is stored. This user data list is a database that manages individual user information and the authority to resolve the exceptional state of the image forming apparatus 101. In this embodiment, the authority to resolve the exceptional state of the image forming apparatus 101 is set according to the user's role.

[0204] Administrator users have the authority to resolve exceptional states in the image forming apparatus 101, while general users are restricted from performing actions to resolve exceptional states. For example, as shown in Figure 26, focusing on number "1", the user with user ID "AAA" is associated with the role of a general user. Similarly, focusing on number "2", the user with user ID "BBB" is associated with the role of an administrator user, and focusing on number "3", the user with user ID "CCC" is associated with the role of a general user. This user data list allows for quick determination of whether a user is a general user or an administrator user.

[0205] The device control program's notification process for device exception status (malfunction) information is similar to the flowchart shown in Figure 16. When the device control program 5000 detects any abnormal status caused by a malfunction, it sends that information to the data conversion control program 1700. The device control program 5000 continues to monitor the status even after sending the malfunction information, unless the user issues a stop command. If the user issues a stop command, it performs a shutdown process and immediately terminates the process. Examples of malfunction information (device abnormality information) include the aforementioned toner depletion and JAM.

[0206] The voice notification processing by the voice control program is the same as the flowchart shown in Figure 19. When the voice control program 600 receives voice data from the data conversion control program 1700, it plays the voice data from the speaker 310. After playing the voice data, there is a period of time during which the program waits for a response from the user. When the voice control program 600 receives a response from the user, or when a certain period of no response has elapsed, it converts the response data and sends it to the data conversion control program 1700. If there is no response, empty data is sent to the data conversion control program 1700. In this case, it is determined that there was "no response".

[0207] <Message generation table> Figure 27 shows an example of a message generation table. The message generation table shown in Figure 27 is a table that shows the relationship between the status of the image forming apparatus 101 (device), the message type, and the user, and is stored in the data management unit 702 of the cloud server 103. In this embodiment, when the image forming apparatus 101 enters an exceptional state where printing is not possible, the recovery procedure is notified only to administrator users who can resolve the exceptional state, and notification of the recovery procedure is restricted to general users. The recovery procedure is notified to administrator users who have business with the image forming apparatus 101 and intentionally approach it.

[0208] As shown in Figure 27, the message generation table contains message types and voice messages for general users, individual users, and administrator users, depending on the state of the image forming apparatus 101. For example, if the image forming apparatus 101 is in an exceptional state such as jam or no K toner remaining, the message type will include at least one of the cause of the malfunction and the appropriate countermeasure for the malfunction. This message is then voice-notified only to administrator users who are capable of resolving the exceptional state.

[0209] Furthermore, if the image forming apparatus 101 is receiving a fax job, the message type will be a message to the individual user indicating that a job exists. This message will be audibly notified only to the individual user. Also, if the image forming apparatus 101 is completing a copy job, the message type will be a message to that effect. This message will be audibly notified to all users, regardless of whether they are general users, individual users, or administrator users.

[0210] As described above, in the notification system 100, the user to whom a message is notified changes according to the status (event) of the image forming apparatus 101. This allows for accurate notification of a message to individuals who are highly relevant to that message. Furthermore, messages are pre-stored in the data management unit 702, which serves as the storage means. This allows for quick notification by selecting a message from the data management unit 702.

[0211] <Overhead view showing the arrangement of the image forming apparatus and imaging apparatus, and the person detection method> Figure 28 is an overhead view showing an example of the arrangement of the image forming apparatus, the imaging device, and the user being monitored. The notification system 100 wants to notify only administrator users who are capable of resolving the exceptional state of the image forming apparatus 101 of the recovery procedure. Therefore, it detects users approaching the image forming apparatus 101 and switches whether or not to notify them of a message depending on the user.

[0212] As shown in Figure 28, it is preferable not to notify a message to a passerby user A who is not moving in a straight line to the installation location of the image forming apparatus 101, even if the user is detected. On the other hand, for users like users B and C in Figure 28 who are moving in a straight line to the installation location of the image forming apparatus 101, it is preferable to proactively send a message notification if user B or user C is an administrator user.

[0213] Whether or not a user is moving linearly towards the image forming apparatus 101 is determined in the same way as in the first embodiment. Face recognition is as described in Figure 17B. User A is not determined to be moving linearly because they are moving in a direction different from the straight line connecting their own position and the installation position of the image forming apparatus 101. Users B and C are determined to be moving linearly because they are moving in the same direction as the straight line connecting their own position and the installation position of the image forming apparatus 101.

[0214] Then, the system determines whether or not to send a message notification depending on whether the target user belongs to the first group. In this way, the notification system 100 can determine whether or not to send a message based on the user's direction of movement. This prevents unnecessary message notifications from being sent if an administrator user is heading towards the monitored area, for example, if that administrator user is in a hurry. This determination is made by using the data analysis unit 5002 as the determination means. Furthermore, in user facial recognition, for example, by further using an infrared sensor, three-dimensional facial information can be used as a feature, improving user identifiability.

[0215] If an administrator user is detected as a passing user, a message is proactively sent to prompt them to perform a degraded setting (degradation operation), similar to the example shown in Figure 20C. This allows an administrator user heading towards the monitored area to temporarily set the image forming apparatus 101 to a usable state, even if that administrator user is in a hurry, provided that the degraded setting is easy to perform. If a service technician, who belongs to the first group, is detected as a passing user, a message is proactively sent to prompt them to replace a part, similar to the example shown in Figure 20D. This allows the service technician to recognize that a part needs to be replaced and, for example, perform the replacement at a later date.

[0216] On the other hand, for general users passing by, proactive message notifications are suppressed because unnecessarily notifying them of parts replacement or functional degradation could cause malfunctions due to unnecessary operations. For such general users, it is preferable to notify them of restricted functions via message only when they approach the image forming apparatus 101 with the intention of using it. Similar to the example shown in Figure 20E, no message notification is sent when a general user passes near the image forming apparatus 101, but a message notifying them of restricted functions is sent when a general user approaches the image forming apparatus 101.

[0217] Furthermore, as a user type, guest users who are not internal users and cannot be individually identified belong to the second group of users, and therefore do not have the authority to operate the image forming apparatus 101. In this case, it is preferable not to notify guest users of messages, not only when they pass by, but also when they approach the image forming apparatus 101. When a guest user passes near the image forming apparatus 101 or approaches the image forming apparatus 101, proactive message notification is suppressed, similar to the example shown in Figure 20F.

[0218] <An example of characteristics used in user type determination> Figure 29 shows an example of the characteristics used when determining user type from images captured by the camera. The notification system 100 notifies a message to a specific user detected within the monitored area, i.e., a user belonging to the first group. Here, we will explain a specific example of facial recognition for determining whether a user detected within the monitored area belongs to the first group.

[0219] As shown in Figure 29, the image forming apparatus 101 has pre-stored an image of the face of an administrator user belonging to the first group, viewed from the front (the face image on the right in the figure). In this state, for example, if the shooting device 106 takes a picture of a user, the face image on the upper left in Figure 29 is obtained as the image of that user's face. This image is a real image of the user's face detected within the monitored area and photographed from the front.

[0220] Then, when performing facial recognition, the positional relationships of the eyes, nose, mouth, etc., are extracted as feature points from the real image using a feature extraction algorithm such as the aforementioned ORB. The degree of agreement between the feature points of the real image and the feature points of the administrator user's face image, which is pre-stored in the image forming apparatus 101, is calculated. If the calculation result is above a predetermined level, the real image is determined to be the face image of the user with user ID "BBB" (see Figure 26). The attributes of the group associated with user ID "BBB" are then determined, and the user is determined to be an administrator user belonging to the first group.

[0221] On the other hand, suppose the lower left face image in Figure 29 is obtained as the user's face image. This image is a real image of the user's face detected within the monitored area, taken from the front. When performing face recognition, the positional relationships of the eyes, nose, mouth, etc., are extracted as feature points from the real image using a feature extraction algorithm such as the aforementioned ORB. The degree of agreement between the feature points of the real image and the feature points of the administrator user's face image, which are pre-stored in the image forming apparatus 101, is calculated. If the calculation result is not above a predetermined level, the real image is determined to be the face image of the user with user ID "AAA" (see Figure 26). The attributes of the group associated with user ID "AAA" are then determined, and the user is determined to be a general user belonging to the second group.

[0222] As mentioned above, the notification system 100 is configured to identify the user (person) in an image by facial recognition based on the image captured by the camera 106. For this facial recognition, machine learning that infers and identifies the user can also be used. In this case, the inference model is preferably a learning model using a neural network whose parameters have been adjusted by, for example, backpropagation. This allows the inference model to perform deep learning, for example, to generate various parameters such as features and weights for learning.

[0223] Furthermore, the machine learning method is not limited to deep learning; any machine learning algorithm such as support vector machines, logistic regression, or decision trees may be used. In this embodiment, the system detects when a user approaches the image forming apparatus 101 based on the image captured by the imaging device 106, but this is not limited to this. For example, a short-range communication means (such as BLE (Bluetooth Low Energy)) capable of detecting when a user approaches the image forming apparatus 101 may be used.

[0224] Furthermore, in this embodiment, the image forming apparatus 101 holds user information and identifies the user, but this is not limited to this, and for example, the cloud server 103 may hold user information and identify the user.

[0225] This embodiment includes the following configurations, methods, and programs. (Configuration 1) An imaging means for capturing an image including a person, An acquisition means for acquiring the role of the person in the image based on the image captured by the imaging means, A determination means that determines the content of a message for a predetermined event according to the role acquired by the acquisition means, A notification system characterized by comprising: a notification means for notifying a message determined by the determination means; (Configuration 2) The notification system according to Configuration 1, characterized in that the acquisition means acquires attribute information relating to the attributes of the person as its role. (Configuration 3) The notification system according to Configuration 1 or 2, characterized in that the notification means has a speaker that notifies the message by voice. (Configuration 4) The predetermined event is a malfunction of a predetermined device, The notification system according to any one of configurations 1 to 3, characterized in that the message includes at least one of the cause of the malfunction and a method for dealing with the malfunction. (Configuration 5) The predetermined event is a malfunction of a predetermined device, The notification system according to any one of configurations 2 to 4, characterized in that the determination means determines, according to the attribute information, the content of the message to be one of the following: a first notification notifying of the suspension or restriction of use of the device due to the malfunction; a second notification notifying of work to degrade the functionality of the device in order to temporarily enable the use of the device even under the malfunction; or a third notification notifying of recovery work to resolve the malfunction. (Configuration 6) The attribute information is information that the person is either a user of the device, an administrator who manages the status of the device, or a maintenance and inspection person who performs maintenance and inspection of the device. The notification system according to configuration 5, characterized in that the decision means determines the first notification for the user, the second notification for the administrator, and the third notification for the maintenance and inspection personnel. (Configuration 7) The notification system according to Configuration 5 or 6, characterized by comprising a storage means for storing the first notification, the second notification, and the third notification in advance. (Configuration 8) The notification system according to any one of Configurations 5 to 7, characterized in that when the notification means notifies a user of the device of the first notification, it is possible to make a first call notification to call an administrator who manages the status of the device, and when the notification means notifies the administrator of the second notification, it is possible to make a second call notification to call a maintenance and inspection person who performs maintenance and inspection of the device. (Configuration 9) The notification system according to Configuration 8, characterized in that the notification means repeatedly passes the first notification and the first call notification, and repeatedly passes the second notification and the second call notification. (Configuration 10) The notification system according to Configuration 8 or 9, characterized by comprising a storage means for storing the first call notification and the second call notification in advance. (Configuration 11) The notification system according to any one of Configurations 2 to 10, characterized in that the determination means re-determines the content of the message each time the attribute information is changed. (Configuration 12) The notification system according to any one of Configurations 2 to 11, characterized in that the notification means repeats the notification of the message and the cessation of the notification at predetermined intervals if there is no change in the attribute information acquired by the acquisition means. (Configuration 13) A notification system according to any one of Configurations 1 to 12, characterized in that it includes a determination means for determining whether or not to notify the message according to the direction of movement of the person. (Configuration 14) The notification system according to any one of Configurations 1 to 13, characterized in that the acquisition means acquires the role using at least one of Oriented FAST and Rotated BRIEF, Speeded-Up Robust Features, and Scale-Invariant Feature Transform. (Configuration 15) The acquisition means identifies the person by performing facial recognition to detect the person's face in the image, and acquires the group to which the person belongs as the role, The notification system according to any one of configurations 1 to 14, characterized in that the notification means notifies the message if the group to which the person belongs is a first group that has predetermined authority over the event, and does not notify the message if the group to which the person belongs is a second group that has authority different from that of the first group. (Configuration 16) The predetermined event is a malfunction of a predetermined device, The notification system according to configuration 15, characterized in that the message includes information about work to degrade the functionality of the device in order to temporarily enable the use of the device even under the malfunction, or information about recovery work to resolve the malfunction. (Configuration 17) The predetermined event is a malfunction of a predetermined device, The notification system according to configuration 15 or 16, characterized in that the first group is a group of administrators who manage the status of the device, and the second group is a group of users who use the device. (Configuration 18) The notification system according to any one of Configurations 15 to 17, characterized in that the notification means notifies the message regardless of whether the identified person belongs to the first group or the second group. (Configuration 19) The predetermined event is a malfunction of a predetermined device, The notification system according to any one of configurations 15 to 18, characterized in that the imaging means captures images of at least the area around the device as the image. (Configuration 20) A detection means for detecting a person, Based on the detection results from the detection means, an acquisition means for acquiring the role of the person, A determination means that determines the content of a message for a predetermined event according to the role acquired by the acquisition means, A notification system characterized by comprising: a notification means for notifying a message determined by the determination means; (Method 1) An imaging process in which an image including a person is captured, An acquisition step to acquire the role of the person in the image based on the image captured in the aforementioned imaging step, A determination step in which the content of a message for a predetermined event is determined according to the role acquired in the acquisition step, A control method for a notification system, characterized by comprising a notification step for notifying the message determined in the aforementioned determination step. (Program 1) A program that causes a computer to execute each of the means of the notification system described in Configuration 1.

[0226] Although preferred embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of its gist. The present invention can also be realized by supplying a program that implements one or more of the functions of the above embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, the present invention can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of symbols]

[0227] 100 Notification System 101 Image forming apparatus (image processing apparatus) 102 Voice control device 103 Cloud Server 106 Imaging device (imaging means) 310 speakers 600 Voice Control Programs 700 System Control Programs 800 Camera Control Program 5000 Device Control Programs 1700 Data Conversion Control Program

Claims

1. A notification system having a speaker that notifies a message by voice, a notification means for outputting a message relating to the first notification from the speaker when a person identified based on an image captured by the imaging means is a person who is a target of the first notification; When the person identified based on the image captured by the imaging means is a person who is a target of a second notification different from the first notification, the notification means outputs a message related to the second notification through the speaker. A notification system characterized by:

2. The notification system described in Claim 1, characterized in that the person who is the subject of the first notification and the person who is the subject of the second notification are each determined by obtaining the role of the person identified based on the image captured by the imaging means based on attribute information regarding the attributes of the person.

3. The notification system described in Claim 1, characterized in that the message related to the first notification and the message related to the second notification are both messages about a specified event, and each message has different content.

4. the predetermined event is a malfunction of a predetermined device; 4. The notification system according to claim 3, wherein the message includes at least one of a cause of the problem and a method of dealing with the problem.

5. the predetermined event is a malfunction of a predetermined device; the first notification is a notification of recovery work to resolve the malfunction, the second notification is a notification of suspension or restriction of use of the device due to the malfunction, or a notification of work to degrade the functionality of the device to temporarily enable use of the device even under the malfunction; The notification system of claim 1 .

6. The person who is the subject of the first notification and the person who is the subject of the second notification are each determined by acquiring the role of the person identified based on the image captured by the imaging means based on attribute information regarding the attributes of the person, the attribute information is information indicating whether the person is a user who uses a predetermined device, a manager who manages the state of the predetermined device, or a maintenance inspector who performs maintenance and inspection of the predetermined device; the notification means issues the first notification to the user and the second notification to the administrator; The notification system of claim 1 .

7. 2. The notification system according to claim 1, further comprising a storage means for storing the first notification and the second notification in advance.

8. 2. The notification system according to claim 1, further comprising a determining means for determining whether or not to output the message depending on a moving direction of a person identified based on the image captured by the imaging means.

9. A method for controlling a notification system having a speaker that notifies a message by voice, comprising: a notification step of outputting a message relating to the first notification from the speaker when a person identified based on an image captured by the imaging means is a person who is a target of the first notification; In the notification step, when a person identified based on the image captured by the imaging means is a person who is a target of a second notification different from the first notification, a message related to the second notification is output by the speaker. A method for controlling a notification system, comprising:

10. A notification system having a computer with a speaker that notifies a message by voice, a program for executing a notification means for outputting a message relating to the first notification from the speaker when a person identified based on an image captured by an imaging means is a person who is a target of the first notification, When the person identified based on the image captured by the imaging means is a person who is a target of a second notification different from the first notification, the notification means outputs a message related to the second notification through the speaker. A program characterized by:

Images