Image processing device, image processing method and program

Abstract

To make it possible to create a panoramic image in which device information is superimposed with high visibility.SOLUTION: After a panoramic image is created using a plurality of images, an object such as an icon indicating a device included in the panoramic image or a character string indicating a model type is superimposed on the created panoramic image. In addition, during the creation of a panoramic image, objects indicating a device included in the images used to create the panoramic image are enlarged to half the size of the image and superimposed on the image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] In recent years, network cameras have become popular, allowing users to remotely control the camera and view images via a network or dedicated line. Network cameras include PT (pan-tilt) models with a rotating camera head that can rotate in the pan (horizontal) and tilt (vertical) directions, as well as models with zoom capabilities. These models allow users to freely change the shooting direction and angle of view remotely. Furthermore, there is a panorama creation function that creates a bird's-eye view image of the monitored area by taking still images while changing the shooting direction of the camera and combining the still images. Patent Document 1 discloses a method for generating a panoramic image from multiple captured images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-60216 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, an overlay function has become known that displays information about slave devices of network cameras present on the captured image by superimposing it on the video. When the overlay function is used with the technology described in Patent Document 1, a panoramic image is created with the slave device information superimposed on the captured image. As a result, the superimposed information becomes small relative to the created panoramic image, making it difficult to recognize the superimposed slave device information. Furthermore, if there is a large amount of superimposed information, the superimposed information appears as noise, resulting in a decrease in the quality of the panoramic image. On the other hand, if no information about the slave device is superimposed, it becomes impossible to know even the location of the slave device on the panoramic image.

[0005] In view of the above-mentioned problems, an object of the present invention is to make it possible to create a panoramic image on which device information is superimposed with high visibility. [Means for solving the problem]

[0006] The image processing device according to the present invention comprises: The apparatus comprises an acquisition means for acquiring a plurality of images taken by moving within a shooting range and information about a device included in at least one of the plurality of images; a superimposition means for superimposing an object corresponding to the information about the device acquired by the acquisition means onto an image of the plurality of images that includes the device; and a creation means for creating a panoramic image from the plurality of images including the image on which the object has been superimposed by the superimposition means, wherein when the panoramic image is created by the creation means, the superimposition means superimposes the object at an enlarged scale compared to when a panoramic image is not generated. [Effects of the Invention]

[0007] According to the present invention, a panoramic image can be created on which device information is superimposed with high visibility. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an example of the internal configuration of a network camera according to an embodiment; [Figure 2] 1 is a diagram illustrating an example of the external configuration of a network camera according to an embodiment. [Figure 3] 4 is a diagram showing an example of the data configuration of an information table held by the network camera according to the embodiment; FIG. [Figure 4] 1 is a diagram illustrating an example of a positional relationship between a network camera and a wireless communication adapter according to an embodiment; [Figure 5] 5 is a flowchart showing an example of processing when panning, tilting, and zooming of the network camera according to the first embodiment. [Figure 6] 10 is a flowchart illustrating an example of a Z-Wave overlay process according to the embodiment. [Figure 7] 10A and 10B are diagrams illustrating an example in which child device information according to the embodiment is superimposed on a captured image. [Figure 8] 10 is a flowchart illustrating an example of a panoramic image creation process according to the embodiment. [Figure 9]FIG. 10 is a diagram illustrating an example of an object to be superimposed according to the embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a created panoramic image according to the embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of a created panoramic image according to the embodiment. [Figure 12] 10 is a flowchart showing an example of processing when panning, tilting, and zooming of a network camera according to the second embodiment. [Figure 13] 10 is a flowchart showing an example of Z-Wave overlay processing during creation of a panoramic image according to the second embodiment. [Figure 14] 10A and 10B are diagrams illustrating an example in which child device information is superimposed on a captured image during creation of a panoramic image according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] (First embodiment) Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations described in the following embodiments. Furthermore, although each embodiment will be described using a network camera having an imaging unit as an example, the present invention can also be applied to imaging devices other than network cameras. For example, the present invention can also be applied to imaging devices having an imaging unit that captures video for broadcasting, movies, or personal video.

[0010] First, the configuration of each part of the network camera 100 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the internal configuration of a network camera, a client device, and a slave device according to an embodiment. The network camera 100 has an image processing unit 102, a system control unit 103, a lens driving unit 104, a lens control unit 105, a pan driving unit 106, a tilt driving unit 107, a pan / tilt control unit 108, a communication unit 109, and a wireless communication control unit 110.

[0011] The system control unit 103 may be provided inside the network camera 100, or may be configured independently of the network camera 100. When the system control unit 103 is an external device to the network camera 100, the system control unit 103 and the network camera 100 may be configured to be able to communicate with each other.

[0012] The imaging unit 101 receives light that has been focused through a lens at an imaging element, converts the received light into electric charges, and generates an imaging signal. The imaging element may be, for example, a CMOS image sensor (Complementary Metal Oxide Semiconductor Image Sensor). Alternatively, the imaging element may be a CCD image sensor (Charge Coupled Device Image Sensor).

[0013] The image processing unit 102 digitizes the image signal generated by the imaging unit 101 to generate image data. At this time, the image processing unit 102 also performs various image processing to correct the image quality. The image processing unit 102 may further compress and encode the image data to generate compressed and encoded image data. Note that the image processing unit 102 may perform these processes by providing a dedicated circuit for some or all of the processes. The communication unit 109 transmits a video stream based on the image data generated by the image processing unit 102 to the client device 200. Note that the image data here is, for example, video image data. The communication unit 109 also receives commands transmitted from the client device 200 and transmits them to the system control unit 103. It also transmits responses to the commands to the client device 200 under the control of the system control unit 103. In this way, the system control unit 103 also functions as a communication control unit.

[0014] Furthermore, the system control unit 103 of the network camera 100 analyzes the command received by the communication unit 109 and performs processing according to the command. For example, the system control unit 103 causes the image processing unit 102 to adjust the image quality in accordance with the command. It issues instructions to the image processing unit 102 to adjust the image quality and instructions to the lens control unit 105 to control zoom and focus. The lens control unit 105 controls the lens driving unit 104 based on instructions transmitted from the system control unit 103 . The lens driving unit 104 is composed of, for example, a driving system for a focus lens and a zoom lens, a motor as a driving source thereof, and the like, and its operation is controlled by a lens control unit 105 . The pan drive unit 106 is composed of, for example, a mechanical drive system that performs panning, a motor as the drive source, an angle sensor that detects the angle of the drive unit, and the like.

[0015] The tilt driving unit 107 is composed of, for example, a mechanical driving system that performs tilting operation, a motor as a driving source thereof, an angle sensor that detects the angle of the driving unit, etc. The operations of the pan driving unit 106 and the tilt driving unit 107 are controlled by a pan / tilt control unit 108. The pan / tilt control unit 108 controls the pan driving unit 106 and the tilt driving unit 107 based on instructions from the system control unit 103 . The wireless communication control unit 110 transmits and receives data, control commands, etc. in accordance with a wireless communication standard, and transmits and receives radio waves that modulate and demodulate the data, control commands, etc. using an antenna 111. The system control unit 103 exchanges data, control commands, etc. with the wireless communication slave unit 400 through the wireless communication control unit 110, and performs processes such as obtaining and setting information for the wireless communication slave unit 400.

[0016] Next, the configuration of each part of the client device 200 will be described with reference to FIG. The client device 200 is configured by a computer such as a personal computer, etc. The client device 200 has a display unit 201, an input unit 202, a system control unit 203, and a communication unit 204. The display unit 201 displays video based on image data received from the network camera 100 and a graphic user interface (hereinafter referred to as GUI) for controlling the camera. The display unit 201 is configured with a display device using a liquid crystal panel, an organic EL panel, or the like. The display on the display unit 201 is performed under the control of the system control unit 203. The input unit 202 is configured with devices such as a keyboard, a mouse, etc. The user of the client device 200 operates the GUI via the input unit 202. The input unit 202 may be configured with a touch panel.

[0017] The system control unit 203 generates commands in response to user operations and transmits them from the communication unit 204 to the network camera 100. The system control unit 203 also displays image data received from the network camera 100 via the communication unit 204 on the display unit 201. In this way, the system control unit 203 also functions as a communication control unit and a display control unit. In this way, the client device 200 can obtain image data from the network camera 100 via the network 300. In addition, the client device 200 can control the network camera 100 by sending commands via the network 300. The communication unit 204 receives image data from the network camera 100. The communication unit 204 also transmits commands to the client device 200.

[0018] Next, the wireless communication slave device 400 will be described with reference to Fig. 1. The wireless communication slave device 400 is an example of a slave device. The wireless communication terminal 400 has, for example, a sensor function, and transmits data acquired by the sensor to the network camera 100 operating as a master terminal. The wireless communication slave device 400 is battery-powered, and when the battery is attached and the device is started up, it performs a registration process with the parent device, and enters a sleep state with limited functionality if no communication is made from the parent device within a predefined communication timeout period. The wireless communication slave device 400 can be registered with the network camera 100 not only when the device is started up by attaching a battery, but also by adding a button to the wireless communication slave device 400 so that the registration process starts when the user presses the button. While the wireless communication slave device 400 is used as a slave device in FIG. 1, this is not a limitation. A slave device connected via a wired connection may also be used. Here, an example of the data configuration of an information table held by the network camera 100 and the positional relationship between the network camera 100 and the wireless communication terminal 400 will be described with reference to FIGS.

[0019] 3 is a diagram showing an example of the data configuration of an information table held by the network camera 100. The information table 500 is held by the system control unit 103, and associates ID 501, device name 502, type 503 indicating the type of device, and pan and tilt 504 indicating the pan and tilt coordinate values ​​within the network camera 100. The pan and tilt 504 are the pan and tilt coordinate values ​​of the network camera 100 when the wireless communication terminal 400 is displayed at the center of the angle of view of the network camera 100 .

[0020] 4 is a diagram showing an example of the positional relationship between the wireless communication slave device 600 and the pan / tilt coordinates of the network camera 100. In this example, the wireless communication slave device 600 is located at the pan / tilt coordinates (45, 45) of the network camera 100. Note that this corresponds to the device with ID "9" in column 505 in FIG. 3.

[0021] Next, an example of the mechanical configuration of the network camera 100 in this embodiment will be described with reference to Fig. 2. Fig. 2(a) is a diagram showing the network camera 100 mounted on a table as seen from above, and Fig. 2(b) is a diagram showing it as seen from the side. In Figs. 2(a) and 2(b), the network camera 100 has a bottom case 121, a turntable 122, a camera head support 123, and a camera head 124.

[0022] Next, the pan / tilt operation will be explained with reference to Figure 2. The axis in the up / down direction is defined as the vertical axis, and the axis perpendicular to this is defined as the horizontal axis. In Figure 2(a), the clockwise direction around the vertical axis perpendicular to the paper surface is defined as the positive direction of the pan angle, and in Figure 2(b), the clockwise direction around the axis perpendicular to the paper surface is defined as the positive direction of the tilt angle. 2, pan drive unit 106 is made up of bottom case 121 and turntable 122, and turntable 122 rotates horizontally around a vertical axis. An electric circuit made up of the fixed part of bottom case 121 and the rotating part of turntable 122 is connected by a slip ring or the like, so pan drive unit 106 can rotate endlessly within an angular range of 360° within a plane including the horizontal axis.

[0023] In this way, the network camera 100 of this embodiment can change the shooting direction and shoot a wide range by rotating the camera head 124 horizontally and vertically. Note that the network camera 100 of this embodiment is not limited to the configuration shown in Figures 2(a) and 2(b). For example, instead of being able to drive 360° in the pan direction, the driveable range may be limited to, for example, +170° to -170°. Furthermore, instead of being able to drive 180° in the tilt direction, the drive range may be limited to, for example, 0° to 90°.

[0024] Next, a processing example when panning, tilting, and zooming are performed on the network camera 100 according to this embodiment will be described with reference to the flowchart in Fig. 5. The processing of this flow may be performed each time panning, tilting, or zooming stops, or may be performed after a certain period of time has elapsed since the stoppage. First, in S701, the system control unit 103 determines whether the current mode is a mode for creating a panoramic image. If the system control unit 103 determines that the current mode is a mode for creating a panoramic image, the process proceeds to S702. On the other hand, if the system control unit 103 determines that the current mode is not a mode for creating a panoramic image, the process proceeds to S703.

[0025] In S703, the system control unit 103 performs Z-Wave overlay processing. Details of the Z-Wave overlay processing will be described later using the flowchart in FIG. In S702, the system control unit 103 performs a process of creating a panoramic image. Details of the process of creating a panoramic image will be described later with reference to the flowchart in FIG. In S704, the system control unit 103 checks the pan and tilt positions and types of the slave units registered in the network camera 100 within the range of the pan and tilt coordinates of the four vertices of the panoramic image.

[0026] In S705, the system control unit 103 superimposes an object indicating the device information of the slave unit at the pan and tilt positions confirmed in S704 on the panoramic image created in S702, and ends the process. Thereafter, the system control unit 103 of the client device 200 receives the captured image (panoramic image) from the network camera 100 and controls the display unit 201 to display it. 9 is a diagram showing an example of objects superimposed in S705. An icon 1101 is a temperature sensor, an icon 1102 is a door sensor, and an icon 1103 is a humidity sensor.

[0027] FIG. 10 is a diagram showing an example of a panoramic image created in S705. Icons 1101 to 1103 are superimposed on a panoramic image 1200. Note that, in FIG. 10, an example has been described in which icons are superimposed as objects indicating device information of the child devices, but this is not limiting. For example, as shown in FIG. 11, a character string indicating the model type of the child device may be superimposed on the panoramic image. In FIG. 11, a character string 1301 indicating a temperature sensor, a character string 1302 indicating a door sensor, and a character string 1303 indicating a humidity sensor are superimposed on the panoramic image 1200.

[0028] Next, the Z-Wave overlay display process in S703 will be described in detail with reference to the flowchart in FIG. In S801, the system control unit 103 acquires the current pan / tilt coordinates from the pan / tilt control unit 108, and further acquires the current zoom from the lens control unit 105. Then, the system control unit 103 calculates the pan / tilt coordinates of the four vertices of the shooting angle of view based on this information. In S802, the system control unit 103 determines whether or not a slave unit is present within the shooting angle of view based on the pan and tilt coordinates calculated in S801 and the information table 500 in Fig. 3. If the system control unit 103 determines that a slave unit is present within the shooting angle of view, the process proceeds to S803. On the other hand, if the system control unit 103 determines that a slave unit is not present within the shooting angle of view, the process of this flow ends. In S803, the system control unit 103 acquires information about the slave units present within the shooting angle of view. Here, the information about the slave units may be acquired by making a status inquiry to the slave units, or information about the slave units that has been saved in advance may be acquired. The information about the slave units includes information about the model of the slave units and information related to the functions of the slave units.

[0029] In S804, the system control unit 103 superimposes the information about the child device acquired in S803 on the captured image, and ends the process. 7 is a diagram showing an example of a captured image on which information about the slave unit is superimposed in this process. Information acquired from the slave unit 901 is displayed in an information display section 902 on a captured image 900. In FIG. 7, the slave unit 901 is a thermometer, and the temperature information of the slave unit 901 is displayed in the information display section 902. When the above Z-Wave overlay processing is completed, the system control unit 103 returns to the flow of FIG. 5 and ends the processing.

[0030] Next, the panoramic image creation process in S702 will be described in detail with reference to the flowchart in FIG. In S1001, the system control unit 103 acquires the current pan / tilt coordinates from the pan / tilt control unit 108, and further acquires the current zoom from the lens control unit 105, and holds these values. In S1002, the system control unit 103 changes the pan, tilt, and zoom values ​​in accordance with the shooting position. In S1003, the system control unit 103 captures a still image. In S1004, the system control unit 103 determines whether or not shooting of all still images in the panoramic image creation area has been completed. If the system control unit 103 determines that shooting of all still images in the panoramic image creation area has been completed, the process proceeds to S1005. On the other hand, if the system control unit 103 determines that shooting of all still images in the panoramic image creation area has not been completed, the process returns to S1002.

[0031] In S1005, the system control unit 103 synthesizes the multiple still images captured by moving the shooting range in S1002 and S1103, i.e., the multiple still images captured by moving the shooting range, to create a panoramic image. At this time, the pan and tilt coordinates of the four vertices of the panoramic image are calculated. In S1006, the system control unit 103 instructs the pan / tilt control unit 108 and the lens control unit 105 to restore the pan, tilt, and zoom values ​​held in S1001. Then, the pan / tilt control unit 108 drives the pan driving unit 106 and the tilt driving unit 107 with the original pan and tilt values, and the lens control unit 105 drives the lens driving unit 104 with the original zoom value. When the above panoramic image creation process is completed, the system control unit 103 advances the process to S704 in FIG.

[0032] As described above, in the first embodiment, the overlay function is temporarily stopped during the panoramic image creation mode, and the device information of the child device is overlaid after the panoramic image is created. This makes it possible to create a panoramic image with device information overlaid with high visibility. In this embodiment, the superimposition processing of S704 and S705 is performed after the panoramic image is created, but if the panoramic image has already been created in S702, the processing of S704 and S705 may be performed when the device information of the slave device is updated. Also, the processing of S704 and S705 may be performed when image data is transmitted to the client device 200.

[0033] (Second embodiment) In the first embodiment, we have explained the case where the overlay function is temporarily stopped while the panoramic image is being created, but it is also possible to change the display mode of the superimposed overlay while the panoramic image is being created. Therefore, in the second embodiment, a processing example in which the display mode of a superimposed overlay is changed while a panoramic image is being created will be described using the flowchart in Fig. 12. Note that a description of the same parts as in the first embodiment will be omitted. The processing of this flow may be performed each time panning, tilting, or zooming stops, or may be performed after a certain time has elapsed since the stoppage.

[0034] First, in S1401, the system control unit 103 determines whether the current mode is a mode for creating a panoramic image. If the system control unit 103 determines that the current mode is a mode for creating a panoramic image, the process proceeds to S1402. On the other hand, if the system control unit 103 determines that the current mode is not a mode for creating a panoramic image, the process proceeds to S703. In S1402, the system control unit 103 performs Z-Wave overlay processing in the panoramic image creation mode. This processing differs from the Z-Wave overlay processing in S703 in the display content and display method. Hereinafter, the Z-Wave overlay process in the panoramic image creation mode will be described in detail with reference to the flowchart of FIG. In steps S1501 to S1503, the system control unit 103 performs the same processes as in steps S801 to S803 in FIG. 6 in the first embodiment.

[0035] In S1504, the system control unit 103 determines an icon for the child device based on the information about the child device acquired in S1503. For example, the model types of the child devices and the corresponding icons are previously associated and stored, and the system control unit 103 determines the icon corresponding to the information about the model type of the child device acquired in S1503 as the icon for the child device. In S1505, the system control unit 103 determines whether the slave unit is located to the right of the center within the shooting angle of view. If the system control unit 103 determines that the slave unit is located to the right of the center within the shooting angle of view, the process proceeds to S1506. On the other hand, if the system control unit 103 determines that the slave unit is located to the left of the center within the shooting angle of view, the process proceeds to S1507.

[0036] In S1506, the system control unit 103 superimposes the icon of the slave device determined in S1504 on the right half of the shooting screen at half the size of the shooting screen. By enlarging and superimposing the icon on the shooting screen in this way, the user can easily recognize the information of the slave device in the panoramic image. In S1507, the system control unit 103 superimposes the icon of the child device determined in S1504 on the left half of the photographed screen, at half the size of the photographed screen. Fig. 14 is a diagram showing an example of the processing result of S1506. An icon 1101 is superimposed on the right half of the shooting screen 1100. Note that Fig. 14 has described an example in which an icon is superimposed as an object indicating device information of the slave device, but this is not limiting. For example, a character string indicating the model type of the slave device may be superimposed. Also, in S1506 and S1507, the icon of the child device is superimposed at half the size of the photographed screen, but the icon may be enlarged to a size that allows the user to recognize the icon in the panoramic image. When the Z-Wave overlay process during creation of the panoramic image described above is completed, the system control unit 103 advances the process to S702 in FIG.

[0037] As described above, in the second embodiment, the display mode of the superimposed overlay is changed during the panoramic image creation mode, thereby making it possible to create a panoramic image on which device information is superimposed with high visibility.

[0038] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0039] 103 System control section

Claims

1. an acquisition means for acquiring a plurality of images captured by moving a photographing range and information about a device included in at least one of the plurality of images; a superimposing means for superimposing an object corresponding to the information about the device acquired by the acquiring means onto an image including the device among the plurality of images; a creating means for creating a panoramic image from the plurality of images including the image on which the object is superimposed by the superimposing means; and 10. An image processing device according to claim 9, wherein, when a panoramic image is created by said creating means, said superimposing means superimposes said object at an enlarged scale compared to when a panoramic image is not generated.

2. 2. The image processing apparatus according to claim 1, wherein the superimposing means enlarges the size of the object to be superimposed to half the size of the image including the device.

3. an acquisition step of acquiring a plurality of images captured by moving a photographing range and information about a device included in at least one of the plurality of images; a superimposing step of superimposing an object corresponding to the information about the device acquired in the acquiring step onto an image including the device among the plurality of images; a creation step of creating a panoramic image from the plurality of images including the image on which the object is superimposed in the superimposing step; and An image processing method characterized in that, when a panoramic image is created in the creating step, the object is superimposed in the superimposing step at a larger scale than when a panoramic image is not generated.

4. an acquisition step of acquiring a plurality of images captured by moving a photographing range and information about a device included in at least one of the plurality of images; a superimposing step of superimposing an object corresponding to the information about the device acquired in the acquiring step onto an image including the device among the plurality of images; a creation step of creating a panoramic image from the plurality of images including the image on which the object is superimposed in the superimposing step; on the computer, When a panoramic image is created in the creating step, the object is superimposed in the superimposing step at a larger scale than when a panoramic image is not generated.

Images