Image processing device, image processing method, and computer program

Abstract

To provide an image processing apparatus or the like that, in specifying an external device, can prevent an adverse effect of emission and flashing of light from an illumination unit of the external device.SOLUTION: An image processing apparatus comprises control means that controls a display screen displaying information on an external device superimposed on moving images including an image of the external device picked up by imaging means. When the control means receives input of an instruction to turn on or flash light from an illumination unit of the external device, if the illumination unit of the external device is in use or an image recognition operation based on the moving images is performed, the control means displays a screen for displaying predetermined specific information near the external device on the display screen in place of turning on or flashing the light from the illumination unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing apparatus, an image processing method, a computer program, and the like. [Background Art]

[0002] Conventionally, image processing apparatuses such as monitoring apparatuses that can monitor video and acquire audio through remote control via a network or a dedicated line have been known. A monitoring apparatus can be connected to an external device to acquire or control information. Furthermore, external devices are generally equipped with an LED (Light Emitting Diode) as an indicator, and for example, the specific external device is identified by the lighting of the indicator.

[0003] Patent Document 1 discloses a display device that lights an indicator provided around its own image display unit with a predetermined brightness when the display device is selected as a communication partner by another image display device.

[0004] Patent Document 2 discloses a system including a user interface that selectively lights illumination of a power indicator when a video device is powered on. [Prior Art Documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent No. 6055421 [Patent Document 2] Japanese Unexamined Patent Application Publication No. 2002-354363 [Summary of the Invention] [Problem to be Solved by the Invention]

[0006] On the other hand, when controlling external devices such as light bulbs via a network, the light bulb itself may also serve as an indicator. In such cases, instructing the indicator to light up or flash may cause the light bulb to light up or flash, potentially affecting the surrounding lighting environment.

[0007] This invention has been made in view of the above-mentioned problems, and aims to provide an image processing device, etc., that can prevent adverse effects from the illumination or blinking of the lighting section of an external device when identifying an external device. [Means for solving the problem]

[0008] The image processing apparatus of the present invention is The system includes control means for controlling a display screen that superimposes information about an external device onto a moving image including the external device captured by the imaging means. The control means is characterized in that, when an instruction to turn on or blink the lighting unit of the external device is received, if the lighting unit of the external device is in use or image recognition operation based on the moving image is in progress, instead of turning on or blinking the lighting unit, it displays a screen on the display screen for displaying predetermined specific information near the external device. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an image processing device, etc., that can prevent adverse effects caused by the illumination or blinking of the lighting section of an external device when identifying an external device. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of the configuration of the monitoring system 10 according to the first embodiment. [Figure 2] (A) is a functional block diagram showing an example of the configuration of the monitoring device 1 according to the first embodiment. (B) is a functional block diagram showing an example of the configuration of the client device 2 according to the first embodiment. [Figure 3]This figure shows an example of the display screen 31 for external device management settings in the monitoring device 1 according to the first embodiment. [Figure 4] This figure shows an example of a display screen 41 for superimposing the "current value" of an external device in the monitoring device 1 according to the first embodiment. [Figure 5] This figure shows an example of a display screen 51 showing an overlaid image according to the first embodiment. [Figure 6] This flowchart shows an example of the processing performed by the monitoring device according to the first embodiment. [Figure 7] This figure shows an example of a display screen that prompts the display of specific information on the display unit of the client device 2 according to the first embodiment. [Figure 8] This figure shows an example of a display screen in which specific information is superimposed using a viewer according to the first embodiment. [Figure 9] This figure shows an example of a display screen 91 for setting image detection of an external device in the monitoring device 1 according to the second embodiment. [Figure 10] This flowchart shows an example of the processing performed by the monitoring device according to the second embodiment. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not limited to the following embodiments. In each drawing, the same reference numeral is used for the same member or element, and redundant explanations are omitted or simplified.

[0012] (First embodiment) Figure 1 shows an example of the configuration of a monitoring system 10 according to the first embodiment, and the monitoring system 10 of this embodiment includes a monitoring device 1, a client device 2, external devices 3a, 3b, etc. In this embodiment, for example, the monitoring device 1 and the client device 2 function as image processing devices.

[0013] The monitoring device 1 is a surveillance camera that captures moving images of subjects such as humans, and for example, a network camera can be used. The monitoring device 1 communicates with a client device 2, external devices 3a and 3b via networks 4 and 5.

[0014] The client device 2 is used to control the operation of the monitoring device 1, and for example, an information processing device such as a PC (personal computer) can be used. The client device 2 includes a display unit 6 and functions as a display control device that controls the display unit 6. The client device 2 also transmits various control commands to the monitoring device 1.

[0015] The control commands include, for example, commands for instructing operation (such as lighting) of the external devices 3a, 3b and the like, commands for instructing lighting or blinking of an indicator for identifying (confirming) the external devices 3a, 3b, commands for image detection setting, commands for controlling the monitoring device 1, and the like. The monitoring device 1 transmits a response to the received control command to the client device 2.

[0016] The external devices 3a and 3b communicate with the monitoring device 1 via the network 5. The monitoring device 1 transmits control commands to the external devices 3a and 3b, and receives notification commands related to device status from the external devices 3a and 3b.

[0017] In the present embodiment, the external devices 3a and 3b are light-type devices, configured to activate an electric circuit in response to a control command received from the outside to light or blink an indicator provided on a housing of the external device. In the present embodiment, the indicator may be an illumination unit such as an LED, or the light itself may be used as the indicator. Here, the indicator and the light function as the illumination unit.

[0018] Further, the external devices 3a and 3b transmit a notification command for notifying the monitoring device 1 of a current state (for example, a current value such as illuminance of the light). The functions of the external devices 3a and 3b are not limited to the function of a light, and they may be devices having other functions such as various sensor devices, for example.

[0019] Monitoring device 1 and client device 2 are connected via network 4, enabling them to communicate with each other. Network 4 consists of multiple routers, switches, cables, etc., that meet a communication standard such as Ethernet®. However, the communication standard, scale, and configuration can be anything as long as communication is possible between monitoring device 1 and client device 2.

[0020] The monitoring device 1 and external devices 3a and 3b are connected in a manner that allows them to communicate with each other via a wireless or wired network 5. Network 5 can use a network based on communication standards such as Z-Wave, Zigbee (registered trademark), or wireless LAN. However, as long as communication is possible between the monitoring device 1 and external devices 3a and 3b, the communication standard, size, and configuration of network 5 may be any.

[0021] When external devices 3a and 3b receive an indicator light-up or flashing command from monitoring device 1, they light up or flash their indicators (lights, LEDs, etc.). One use for lighting up or flashing the indicators is to identify (confirm) which of the installed external devices is being operated on the display screen. However, the use of the indicators is not limited to this; for example, they can also be used to check the operating status of external devices 3a and 3b.

[0022] In this embodiment, an example of blinking the indicator will be described later, but the indicator may also be made to stay lit. If the communication standard of network 5 is Z-Wave, the indicator blinking instruction command should be the Indicator Set command or the like. The blinking method of the indicator, such as the intensity of light emission, blinking time, number of blinks, and the ratio of light emission to off time in one blink, may be configured by the indicator blinking instruction command, or it may blink according to a blinking method predetermined by an external device.

[0023] Next, Figure 2(A) is a functional block diagram showing an example of the configuration of the monitoring device 1 according to the first embodiment, and (B) is a functional block diagram showing an example of the configuration of the client device 2 according to the first embodiment.

[0024] Furthermore, some of the functional blocks shown in Figures 2(A) and (B) are implemented by having a computer included in the monitoring device 1 or client device 2 execute a computer program stored in a storage unit acting as a storage medium. However, some or all of these may also be implemented in hardware.

[0025] In terms of hardware, dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs) can be used. Furthermore, the respective functional blocks shown in Figures 2(A) and (B) do not necessarily have to be built into the same enclosure, and may be composed of separate devices connected to each other via signal paths.

[0026] As shown in Figure 2(A), the monitoring device 1 includes a control unit 11, a storage unit 12, an imaging unit 13, a compression encoding unit 14, a communication unit 15, an imaging control unit 16, and the like. Each component of the monitoring device 1 is connected via a bus 17.

[0027] As mentioned above, the control unit 11 has a built-in CPU as a computer and controls the operation of each part of the entire monitoring device 1 based on a computer program stored in the memory as a storage medium. Furthermore, the control unit 11 functions as a control means for controlling a display screen that overlays information about external devices onto moving images including those devices captured by the imaging unit 13, which is an imaging means.

[0028] The memory unit 12 stores various types of data. The memory unit 12 mainly contains a storage area for computer programs executed by the CPU of the control unit 11 and a work area for programs currently being executed. The memory unit 12 is also used as a storage area for various types of data, such as management information for external devices, setting information for image detection, rule setting information for event triggers and actions, and a storage area for image data generated by the imaging unit 13 and the control unit 11.

[0029] Furthermore, when the monitoring device 1 adds an external device to its management scope, it acquires various information about the external device, such as supported command information and the device type of the external device, and stores it in the storage unit 12.

[0030] The imaging unit 13 captures an image of a subject formed by an imaging optical system (not shown) and acquires an analog signal. The analog signal is then converted into digital data, necessary image processing is performed, and the captured image is sequentially stored in the storage unit 12. The control unit 11 receives an image acquisition event from the imaging unit 13 when the captured image is stored in the storage unit 12.

[0031] The compression encoding unit 14 generates image data by performing compression encoding processing on the captured image or superimposed image stored by the imaging unit 13 based on a format such as JPEG (Joint Photographic Experts Group) or H.264, and stores it in the storage unit 12.

[0032] The communication unit 15 receives control commands from an external device such as the client device 2 and transmits a response to the control command to the external device. When the communication unit 15 receives a command from the external device, the control unit 11 receives a command reception event from the communication unit 15. The communication unit 15 also transmits the control command generated by the control unit 11 to the external device. Furthermore, the communication unit 15 sequentially reads the image data compressed and encoded by the compression encoding unit 14 from the storage unit 12 and transmits it to the client device 2 as a video stream.

[0033] The imaging control unit 16 changes the field of view of the imaging unit 13 by driving the drive units to pan, tilt, zoom, rotate, or focus, respectively, based on the pan, tilt, rotation, zoom, or focus values ​​instructed by the control unit 11. The monitoring device 1 may not include some or all of the above drive units. Alternatively, pan, tilt, rotation, zoom, etc., may be performed electronically by cropping the image.

[0034] Furthermore, as shown in Figure 2(B), the client device 2 includes a control unit 21, a storage unit 22, an operation unit 24, a communication unit 25, a display control unit 26, and the like. Each component of the client device 2 is connected via a bus 27.

[0035] The control unit 21 has a built-in CPU, which functions as a control means to control the operation of each part of the entire client device 2 based on a computer program stored in the memory, which serves as a storage medium.

[0036] The memory unit 22 and the communication unit 25 have the same configuration as the memory unit 12 and the communication unit 15, respectively, and the operation unit 24 includes a keyboard, mouse, and touch panel. 26 is the display control unit, which controls the images displayed on the display unit 6. 27 is the bus.

[0037] Next, an example of a display screen shown on the display unit 6 of the client device 2 will be described. Figure 3 is a diagram showing an example of the display screen 31 for external device management settings in the monitoring device 1 according to the first embodiment. The display screen 31 is displayed when the CPU, which acts as a computer in the control unit 21 of the client device 2, executes a program stored in the storage unit 22 of the client device 2 based on an instruction from the user to perform external device management settings.

[0038] The display screen 31 shows management information 311 and 312, which display the management information for external devices 3a and 3b respectively, an add button 313 for adding a new external device, and a delete button 314 for deleting a registered external device.

[0039] The user presses the add button 313 to switch the monitoring device 1 to a state where external devices can be added. At the same time, by pressing a designated physical button on the casing of the external device to be added, the user can switch that external device to a state where it can be added, thereby registering that external device as an external device managed by the monitoring device 1.

[0040] The method of registering external devices is not limited to this; for example, external devices may be registered by other methods, such as automatically registering them based on identification information of external devices that have been previously registered in monitoring device 1. In this way, external devices 3a and 3b are registered in monitoring device 1 in advance, and management information 311 and 312 are displayed.

[0041] The management information 311 displays the "device number" to identify the registered device, the "type" indicating the main function of the external device 3a, and the "operating status" indicating whether communication with the external device 3a is enabled or disabled. Furthermore, it displays the "current value" representing the current status of the external device's functions (e.g., current operating rate). This information is transmitted from the external device 3a, received by the communication unit 15, and stored in the storage unit 12.

[0042] In the management information 311 in Figure 3, "Equipment Number" is set to "2", "Type" to "Lighting Equipment" indicating a light-type device, "Operating Status" to "On" indicating that it is in operation, and "Current Value" to "80%", which is the operating rate of the luminous intensity. However, the information displayed in the management information 311 is not limited to the information described above, and any information related to the external equipment 3a is included, regardless of its type or combination.

[0043] Furthermore, the "Type" field may display something like "Multi-level switch" to indicate that it is not a lighting fixture but a device that allows for numerical setting. In addition, the management information 311 displays a button labeled "Indicator Flashing" which makes the indicator of external device 3a flash in order to identify external device 3a.

[0044] The management information 312, which displays information about external device 3b, shows "3" as the "device number," and the other information is the same as that in management information 311.

[0045] Figure 4 shows an example of a display screen 41 for superimposing the "current value" of an external device in the monitoring device 1 according to the first embodiment. The display screen 41 is displayed when the CPU, which acts as a computer in the control unit 21 of the client device 2, executes a program stored in the storage unit 22 of the client device 2 based on an instruction from the user to set the superimposed display.

[0046] The entire display screen 41 shows the image capture screen. In addition, the display screen 41 also shows external devices 3a and 3b, an add button 411 for adding a new superimposed display, a delete button 412 for removing a superimposed display, the subject 413, and superimposed display setting frames 414 and 415 that show the current values ​​of external devices 3a and 3b.

[0047] The user displays a superimposed display on screen 41 showing the current values ​​of external devices that were previously added on screen 31. To do this, clicking the add button 411 displays a display frame, such as the superimposed display setting frame 414, on screen 41. Next, by specifying the device number of the external device to be associated with the superimposed display setting frame 414, the current values ​​obtained from the associated external device are displayed within the superimposed display setting frame 414.

[0048] The superimposed display setting frame 414 displays, for example, the current value of 80% obtained from external device 3a. Considering that the external device may have multiple functions, the configuration may specify the type of function to display in addition to the device number, or it may display the current values ​​of multiple functions separately. Furthermore, it may display other values ​​besides the "current value" of the external device's function, such as "drive status." Additionally, settings may be added to specify the pan and tilt angles and zoom magnification range for displaying the superimposed display.

[0049] Next, using the mouse or other operating unit 24 of the client device 2, the overlay display setting frame 414 is dragged and dropped on the display screen 41 to position it near the corresponding external device 3a. Specifically, it is positioned near the corresponding external device (external device 3b in the case of the overlay display setting frame 415), as in the case of the overlay display setting frame 415.

[0050] The information obtained in this way, such as the setting status of the superimposed display setting frame, is stored in the storage unit 12 of the monitoring device 1. Based on the stored information on the superimposed display setting frame, the control unit 11 generates an image (video) with the superimposed display superimposed on the captured image, and stores it as a superimposed image (superimposed video) in the storage unit 12 or storage unit 22.

[0051] Additionally, you can delete the overlay display setting frame by selecting it on the screen using the mouse or other means and clicking the delete button 412.

[0052] Figure 5 shows an example of a display screen 51 displaying a superimposed image according to the first embodiment. The display screen 51 is displayed by executing a program stored in the storage unit of the client device 2 in response to the client device 2 issuing an instruction to display the display screen.

[0053] The display screen 51 in Figure 5 shows superimposed displays 511 and 512 of information about the external devices (such as the illuminance of the lighting unit) corresponding to the external devices 3a and 3b set on the display screen 41, as well as the subject 413.

[0054] Next, an example of the operation of the monitoring device 1 will be explained with reference to the flowchart in Figure 6. Figure 6 is a flowchart showing an example of the processing of the monitoring device according to the first embodiment, and the flowchart in Figure 6 is realized by the CPU in the control unit 11 of the monitoring device 1 executing a program stored in the storage unit 12.

[0055] In step S61, the control unit 11 receives a command to blink the indicator from the client device 2 via the communication unit 15. At this time, the user clicks the indicator blinking buttons 315 or 316 in the management information for the external device they want to identify from among the external devices displayed on the display screen 31.

[0056] When the button is clicked, a command to instruct the indicator to blink is sent from the client device 2, received by the communication unit 15 of the monitoring device 1, and stored in the storage unit 12. In this flowchart example, it is assumed that the user clicks the indicator blinking button 315 of the management information 311 to identify, for example, an external device 3a.

[0057] In step S62, the control unit 11 determines whether the external device that triggered the indicator flashing instruction is a light-type device. Specifically, the control unit 11 obtains information about the external device that triggered the indicator flashing instruction from the storage unit 12 and determines whether it is a light-type device.

[0058] In this case, if the communication standard for network 5 is, for example, Z-Wave, and the Device Type included in the information is determined to be a Light Dimmer Switch or Color Switch, then it is determined to be a light-type device. Alternatively, if the external device supports the Color Switch command class, it is determined to be a light-type device.

[0059] Alternatively, a database file containing the device types of various external devices may be stored in the storage unit 12 beforehand, and the system may compare the target external device with the database file to determine whether or not it is a light-type device. If it is determined to be a light-type device, the system proceeds to step S64; otherwise, it proceeds to step S63. In the example in Figure 6, the indicator blinking button 315 of the management information 311 is clicked to identify the external device 3a, so the system proceeds to step S64.

[0060] In step S63, the control unit 11 sends an indicator blinking command from the communication unit 15 to the external device that received the indicator blinking instruction in step S61, and terminates the processing of this flowchart. For example, if the communication standard of network 5 is Z-Wave, the indicator blinking instruction command is the Indicator Set command or the like. As a result, the indicator (not shown) on external device 3a blinks.

[0061] In step S64, the control unit 11 determines whether the light (illumination unit) of the target external device is currently lit. Specifically, the control unit 11 sends a command to the target external device to check the current value of the light, and makes a determination based on the content of the reply. At this time, the control unit 11 determines that the illumination unit is in use if the illumination unit is lit.

[0062] In this embodiment, the communication standard for network 5 is assumed to be Z-Wave, for example. Therefore, the Current Value or Target Value of the Multi-level Switch Report, Binary Switch Report, or Color Switch Report included in the information obtained from the external device is checked. If these values ​​are 0x01 or greater, it is determined that the light is currently on.

[0063] Alternatively, the value obtained when a lighting command was previously issued to the target external device may be stored in the storage unit 12 as the current value of the light, and it may be determined whether the light is currently lit based on that current value. If it is determined that the light is currently lit, the process proceeds to step S65; otherwise, the process proceeds to step S63.

[0064] In step S65, the control unit 11 determines whether or not to display a mark or the like on the display screen of the client device 2 to identify the external device. To do this, the control unit 11 sends a command to the client device 2 via the communication unit 15 to display the display screen 71 of Figure 7, and the display screen 71 is displayed on the display unit 6 of the client device 2.

[0065] In other words, step S65 functions as a control step that controls a display screen that superimposes information about the external device onto a moving image including the external device captured by the imaging means. In step S65, a screen for displaying predetermined specific information is displayed near the external device on the display screen. Here, the screen for displaying specific information includes a screen that prompts the display of specific information (display screen 71 in Figure 7).

[0066] Figure 7 shows an example of a display screen that prompts the display of specific information on the display unit of the client device 2 according to the first embodiment. The display screen 71 shown in Figure 7 is displayed when the CPU in the control unit of the client device 2 executes a program stored in the storage unit of the client device 2 in response to a command sent from the control unit 11 in step S65.

[0067] On display screen 71, since blinking the indicator of the selected external device would affect the lighting environment, a dialog box is displayed to ask whether to display identifying information (such as a mark) on the viewer screen to identify the external device instead. Display screen 71 also displays a button 711 to decide whether to display identifying information (such as a mark) on the viewer screen to identify the external device, and a button 712 to decide whether to not display identifying information on the viewer screen to identify the external device.

[0068] Furthermore, the control unit 11 may check whether the setting for displaying specific information (such as a mark) to identify the target external device has been set, by looking at the specific information display settings stored in the storage unit 12. If the specific information display settings are not stored, the control unit 11 may instruct the client device 2 to display a warning asking whether the flashing of the light will affect the lighting environment and whether the flashing should still be done. In other words, the screen prompting the display of specific information may include a screen that displays a warning, for example.

[0069] The display screen 71 is shown, and when the user clicks button 711 or button 712, decision information is sent from the communication unit of the client device 2 to the monitoring device 1, and the decision information received via the communication unit 15 is stored in the storage unit 12.

[0070] In step S65, it is determined whether the received decision information is a decision to display specific information on the viewer screen. If button 712 is clicked and the result in step S65 is determined to be No (i.e., the specific information display setting is not selected), the indicator flashes in step S63 and then this flowchart is terminated.

[0071] In other words, if the lighting unit of the external device is not in use, the lighting unit will be turned on or blinked in step S63. Alternatively, if button 711 in Figure 7 is clicked on the client device, the client device may proceed directly to display screen 81.

[0072] If the answer in step S65 is Yes, the process proceeds to step S66, where the control unit 11 determines whether the superimposed display of information about the target external device (such as the illuminance of the lighting unit) is included within the field of view captured by the imaging unit 13. Due to the field of view change by the imaging control unit 16, the superimposed display of the target external device set on the display screen 41 may be outside the field of view. In other words, depending on the pan-tilt angle and zoom magnification being captured by the monitoring device 1, the superimposed display of the external device may not be included within the field of view (within the screen).

[0073] Therefore, in step S66, the control unit 11 determines whether all coordinates of the superimposed display of information about the target external device stored in the memory unit 12 are included within the field of view of the current captured image. Furthermore, even if the superimposed display is within the field of view of the captured image, if a setting is made to limit the range of the field of view in which the superimposed display is actually shown, the control unit 11 determines that the superimposed display of information about the target external device is included within the field of view only when the field of view is within that range. This is because there may be settings that allow the superimposed display to be shown only within a specific range of pan-tilt angles or zoom magnification.

[0074] If it is determined in step S66 that the field of view includes an overlaid display of information about the target external device, proceed to step S68; otherwise, proceed to step S67. Also, if the monitoring device 1 does not support changes in the field of view, proceed to step S68.

[0075] In step S67, the control unit 11 changes the field of view so that the superimposed display is displayed within the field of view of the captured image. Specifically, the control unit 11 instructs the imaging control unit 16 to change the field of view so that all coordinates of the superimposed display of information about the target external device stored in the memory unit 12 fall within the field of view in which the superimposed display is displayed. In other words, the field of view is changed so that the superimposed display of information about the external device is displayed within a predetermined field of view of the moving image.

[0076] The instruction to change the field of view can be made in any way, as long as all the coordinates of the superimposed display are included within the field of view in which the superimposed display is shown. For example, the field of view when the superimposed display was set on the display screen 41 can be stored in the memory unit 12 and an instruction to change the field of view can be made to match the stored field of view, or an instruction to change the field of view can be made so that the superimposed display is displayed in the center of the screen.

[0077] In step S68, the control unit 11 overlays and displays identification information of the target external device on the captured image as shown in Figure 8. Figure 8 is a diagram showing an example of a display screen with identification information overlaid in the viewer according to the first embodiment. As shown in Figure 8, the control unit 11 generates a superimposed image by overlaying identification information 811 (icons, marks, etc.) for identifying the target external device onto the captured image. That is, it displays a screen (display screen 81 in Figure 8) for displaying predetermined identification information near the external device.

[0078] The video stream based on the generated superimposed image is distributed to the client device 2 via the communication unit 15, and the display unit 6 of the client device 2 displays a viewer screen like the display screen 81 in Figure 8.

[0079] In other words, the entire display screen 81 shows the image capture screen, and in addition, it displays the external devices 3a and 3b, the subject 413, an overlaid display 512 corresponding to the current value of the external device 3b set on the display screen 41, and specific information 811 (icons, marks, etc.) that identifies the external device 3a.

[0080] Furthermore, when generating the superimposed image, the control unit 11 superimposes specific information 811, such as an icon or mark that identifies the external device 3a, at the position of the superimposed display 511 in Figure 5, instead of the current value of the external device 3a, or as additional information. The generated superimposed image (superimposed video) is then stored in the storage unit 12. The specific information 811 may be, for example, an icon or mark representing a shape such as a star, circle, or square, or it may be an icon or mark of any shape or color.

[0081] Alternatively, information about external devices (such as device number, device name, or current value) may be displayed prominently as specific information. That is, the superimposed specific information (such as device number, name, mark, or current value) may be displayed by periodically flashing, changing its size, or changing its color or brightness, thereby highlighting it to make it stand out. In other words, the information about external devices as specific information may include at least one of the external device's device number, external device name, or current value relating to the external device's operating status.

[0082] Furthermore, the way the highlighting is displayed may be changed according to the operating status of the external device, such as changing the color to green when the external device is on, yellow when it is in sleep mode, and red when it is off. Additionally, the color of specific information may be changed to be darker or brighter when the average brightness around that information is high or low.

[0083] In this embodiment, an example was described in which specific information is superimposed on the image being captured by the imaging unit 13. However, for example, a snapshot image taken when the superimposition display setting for the target external device is performed on the display screen 41 may be stored in the storage unit 12, and that snapshot image may be used for the superimposition display of the specific information.

[0084] Specifically, if the control unit 11 determines in step S65 that a decision has been made to identify an external device on the viewer screen, it proceeds to step S68. When generating a superimposed image with the identified information superimposed in step S68, the identified information may be superimposed on the aforementioned snapshot image at the position specified in the superimposed display settings for the target external device.

[0085] As described above, in the first embodiment, when the user clicks the indicator flashing button on the display screen 31 for an external device they wish to identify, if the light on the target external device is on, information identifying the target external device (such as a mark) is superimposed on the image.

[0086] In other words, in the image processing method shown in steps S61 to 68, when an instruction to turn on or blink the lighting unit of an external device is received, the control unit 11 determines whether or not the lighting unit of the external device is in use. If it is in use, instead of turning on or blinking the lighting unit, it displays a screen on the display screen that shows predetermined specific information near the external device.

[0087] Therefore, even when the light fixture is on and illuminating the subject while monitoring, external devices can be identified and confirmed on the viewer screen without the light fixture flashing. Furthermore, if an indicator light is used instead of a light fixture, a high intensity of the indicator light may have adverse effects; however, identifying and confirming external devices on the viewer screen can mitigate these adverse effects.

[0088] (Second embodiment) In the first embodiment, the case of determining whether or not the light of an external device is on was described, but in the second embodiment, the case of determining whether or not image detection is set is described. The configuration of the monitoring system according to this embodiment is the same as in the first embodiment.

[0089] Figure 9 shows an example of a display screen 91 for setting image detection of an external device in a monitoring device 1 according to a second embodiment. The display screen 91 is displayed when the CPU in the control unit of the client device 2 executes a program stored in the storage unit of the client device 2 based on an instruction to set image detection.

[0090] The display screen 91 shows external devices 3a and 3b, an add button 911 for adding a new image detection area, a delete button 912 for deleting an added image detection area, an image detection area 913, and superimposed displays 511 and 512 that show the current values ​​of external devices 3a and 3b.

[0091] The user pre-configures image detection settings for the moving images captured by the imaging unit 13. Image detection can be any method that analyzes and detects moving objects in the moving image, such as motion detection that detects moving objects within a detection area. When the user clicks the add button 911, a setting frame for the detection area, such as the image detection area 913, is displayed.

[0092] Afterward, using an input device such as a mouse connected to client device 2, the user can drag and drop the setting frame to move the image detection area to an appropriate position on the screen or change its size. To delete a setting frame, the user can click on the setting frame and then click the delete button 912.

[0093] Furthermore, image detection can be set to apply to the entire screen, or by drawing a boundary line to define the detection boundary, or in any other way depending on the type of image detection. The resulting image detection settings are stored in the storage unit 12 via the communication unit 15 of the monitoring device 1 from the communication unit 2 of the client device 2.

[0094] Next, an example of processing when image detection is set will be explained with reference to a portion of the flowchart in Figure 10. Figure 10 is a flowchart of an example of processing of a monitoring device according to the second embodiment. Note that the CPU, which acts as a computer within the control unit 11, executes a computer program stored in memory to perform each step of the flowchart in Figure 10.

[0095] In the flowchart of Figure 10, the steps with the same code numbers as those in the flowchart of Figure 6 perform the same processing, so their explanation is omitted. In Figure 10, instead of the process of determining whether the light is on or not, in step S101, the control unit 11 determines whether the image detection setting is enabled or not.

[0096] Specifically, the control unit 11 obtains from the storage unit 12 whether or not image detection is enabled on the display screen 91. If it determines that image detection is enabled, the process proceeds to step S65; otherwise, the process proceeds to step S63.

[0097] Furthermore, in addition to setting image detection, additional conditions may be set. For example, in step S101, the control unit 11 refers to the storage unit 12 to determine whether image detection is set and whether a rule is set to perform a predetermined action triggered by an image detection event. If the answer is Yes, the system proceeds to step S65; if No, it proceeds to step S63. That is, if the system is not currently performing an image recognition operation based on moving images, the lighting unit is turned on or blinked in step S63.

[0098] Furthermore, if an image detection event has occurred, it may be undesirable to cause a change in the lighting environment because some event, such as the passage of a moving object, is currently occurring. Therefore, the control unit 11 may refer to the storage unit 12 to check if an image detection event has occurred, and proceed to step S65 if one has occurred, and to step S63 otherwise.

[0099] In this second embodiment, when the user clicks the indicator flashing button on the display screen 31 for an external device they wish to identify, if image detection is enabled, information identifying the target external device (such as a mark) is superimposed on the image.

[0100] In other words, in the second embodiment, in steps S61 to 68 of Figure 10, the control unit 11 determines whether or not an image recognition operation based on moving images is in progress when an instruction to turn on or blink the lighting unit of the external device is received. If an image recognition operation is in progress, instead of turning on or blinking the lighting unit, the control unit 11 displays a screen on the display screen that shows predetermined specific information near the external device.

[0101] Therefore, external devices can be identified on the viewer screen without the indicator flashing, and changes in the lighting environment caused by the flashing indicator can prevent malfunctions in the image detection operation.

[0102] (Other embodiments) Although the present invention has been described in detail above based on preferred embodiments (examples), the present invention is not limited to the above embodiments (examples), and various modifications are possible in accordance with the spirit of the present invention, and these modifications are not excluded from the scope of the present invention.

[0103] For example, some or all of the functional configuration of the monitoring system described above may be implemented as hardware in monitoring device 1, client device 2, and external devices 3a and 3b.

[0104] Furthermore, the configurations and processes of each embodiment described above may be applied to computers and other devices other than the monitoring device 1, client device 2, and external devices 3a and 3b. That is, the processes and operations described in Figures 3 to 10 may be performed by the control unit 11 of the monitoring device 1, by the client device 2, by other devices, or jointly by the monitoring device 1, client device 2, and other devices. Accordingly, the image processing apparatus of this embodiment includes the monitoring device 1, client device 2, and other devices.

[0105] Furthermore, although the external devices constituting the monitoring system of the above-described embodiment are referred to as external devices 3a and 3b for explanatory purposes, there is no limit to the number of external devices as long as there is one or more.

[0106] Furthermore, the monitoring device 1 may implement at least a part of its configuration using hardware. In the case of hardware implementation, for example, a dedicated circuit can be automatically generated on an FPGA (Field Programmable Gate Array) from a program to implement each step by using a predetermined compiler.

[0107] Alternatively, the gate array circuit may be formed in a similar manner to an FPGA and implemented as hardware. Alternatively, it may be implemented using an ASIC (Application Specific Integrated Circuit).

[0108] Furthermore, the present invention can also be realized by performing the following process: supplying a program that realizes the functions of the above-described embodiment to a system or device via a network or various recording media, and having the computer (CPU, MPU, etc.) of that system or device read and execute the program code. In this case, the program and the recording media storing the program constitute the present invention. [Explanation of Symbols]

[0109] 1: Monitoring device 2: Client device 3a, 3b: External equipment 11: Control Unit 12: Storage part 13: Imaging Unit 14: Compression Encoding Section 15: Communications Department 16: Imaging Control Unit 17: Bus

Claims

1. The system includes control means for controlling a display screen that superimposes information about an external device onto a moving image including the external device captured by the imaging means. The control means is characterized in that, when an instruction to turn on or blink the lighting unit of the external device is input, if the lighting unit of the external device is in use or image recognition operation based on the moving image is in progress, instead of turning on or blinking the lighting unit, it displays a screen on the display screen for displaying predetermined specific information near the external device.

2. The image processing apparatus according to claim 1, characterized in that the control means determines that the illumination unit is in use when the illumination unit is lit.

3. The image processing apparatus according to claim 1, characterized in that the specified information is a prominently highlighted representation of information relating to the external device displayed on the display screen.

4. The image processing apparatus according to claim 3, characterized in that the information relating to the external device includes at least one of the device number of the external device, the name of the external device, and a current value relating to the operating status of the external device.

5. The image processing apparatus according to claim 1, characterized in that the specified information includes a predetermined icon or mark.

6. The image processing apparatus according to claim 1, characterized in that the control means changes the field of view so that the specific information is displayed within a predetermined field of view of the moving image.

7. The image processing apparatus according to claim 1, characterized in that the screen for displaying the specified information includes a screen prompting the display of the specified information.

8. The image processing apparatus according to claim 7, characterized in that the screen prompting the display of the aforementioned specific information includes a screen that displays a warning.

9. The image processing apparatus according to claim 1, characterized in that the control means causes the lighting unit of the external device to light up or blink when an instruction to light up or blink the lighting unit of the external device is received, unless the lighting unit of the external device is in use or an image recognition operation based on a moving image is in progress.

10. The image processing apparatus according to claim 1, characterized in that the illumination unit includes an indicator.

11. The image processing apparatus according to claim 1, characterized in that the illumination unit includes an electric light.

12. The system includes a control step for controlling a display screen that superimposes information about an external device onto a moving image including the external device captured by the imaging means. The control step is an image processing method characterized in that, when an instruction to turn on or blink the lighting unit of the external device is input, if the lighting unit of the external device is in use or image recognition operation based on the moving image is in progress, instead of turning on or blinking the lighting unit, the display screen displays a screen for displaying predetermined specific information near the external device.

13. A computer program for causing a computer to function as one of the means of an image processing apparatus described in any one of claims 1 to 10.

Images