Control unit, control method, and program

Abstract

To appropriately change the range of a cut-out image even when a plurality of partial images is cut out from a picked-up image.SOLUTION: When processing is performed which is to change at least one of the position and the size of a first partial image to be cut out from a picked-up image picked up by imaging means, determination means determines whether to change the angle of view of the picked-up image. When the determination means determines to change the angle of view of the picked-up image, angle of view control means performs control to change the angle of view of the picked-up image to include the area of the first partial image, the position or the size of which is changed through the processing. Cut-out control means performs control to change at least one of the position and the size of a second partial image, which is cut out from the picked-up image and which is different from the first partial image, relative to the angle of view of the picked-up image after the change.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a control technique for imaging images.

Background Art

[0002] In recent years, a pan-tilt-zoom camera (hereinafter also referred to as a PTZ camera) that can remotely drive the imaging direction of a camera in the pan direction or the tilt direction has been used in video production sites and surveillance cameras. At the time of video production, an external control device called a controller is connected to the PTZ camera by a cable, and the stick provided on the controller is operated to remotely perform pan, tilt, and zoom operations of the PTZ camera, and the imaging direction of the camera is changed to image a subject.

[0003] In addition, many cameras having a crop function for cutting out and outputting an arbitrary range from an imaging image are also used. In a camera corresponding to the crop function, a user can register an arbitrary range from the imaging image as a crop range (partial image). The camera cuts out and resizes the partial image of the specified range from the imaging image, and outputs it to an external device via a video output terminal such as HDMI (registered trademark) or SDI (Serial Digital Interface), or a network.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] When changing the range of the crop area in the crop function, the user can only specify within the range of the captured image. Therefore, when it is desired to change at least one of the position and size of the crop area beyond the captured range, it is necessary to widen the captured range by performing a zoom-out operation once, which makes the operation cumbersome. Also, when changing the zoom, the angle of view of other already set crop areas will shift, resulting in having to re-set them.

[0006] In Patent Document 1, in the UI (User Interface) for controlling zoom, when the region of interest exists at the boundary of the zoom range where it is specified, a technique is proposed to adjust the zoom range so that the captured range includes the region of interest. However, this technique does not consider the case where there are multiple regions of interest.

[0007] In Patent Document 2, a technique is disclosed for changing the cut-out position of digital zoom according to the drive of pan-tilt. However, this technique assumes that the range to be cut out is always within the range of the captured image, and it does not drive the pan-tilt zoom so that the cut-out position fits.

[0008] Therefore, an object of the present invention is to enable appropriate change of the range of the cut-out image even when a plurality of partial images are cut out from the captured image.

Means for Solving the Problems

[0009] In order to solve the above problems, the control device of the present invention has the following configuration. That is, when a process of changing at least one of the position and size of a first partial image cut out from a captured image captured by an imaging unit is performed, determination means for determining whether to change the field angle of the captured image, and when it is determined by the determination means that the field angle of the captured image is to be changed, angle-of-view control means for performing control to change the field angle of the captured image so as to include the region of the first partial image after being changed by the process, and when the field angle of the captured image is changed by the angle-of-view control means, cutting control means for performing control to change at least one of the position and size with respect to the changed field angle of the captured image in a second partial image cut out from the captured image and different from the first partial image.

Effect of the Invention

[0010] According to the present invention, even when a plurality of partial images are cut out from a captured image, it is possible to appropriately change the range of the cut-out image.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the embodiments described below are examples of means for realizing the present invention. It should be appropriately modified or changed according to the configuration and various conditions of the device to which the present invention is applied, and the present invention is not necessarily limited to the following embodiments. Also, not all combinations of the configurations described in the embodiments are essential for the solution means of the present invention. In addition, in the following, the meaning of the term "video" is not necessarily limited to a moving image, and may be a still image.

[0013] Also, in the following description, embodiments for changing the position and size of the captured image and the crop range will be described. Depending on the situation, at least one of the position and size of the captured image and the crop range may be changed. That is, depending on the situation, it is not necessary to change both the position and size of the crop range. Also, depending on the situation, it is not necessary to change both the position and size of the captured image. In the following description, when expressed as "position·size", it is intended to mean "at least one of the position and size".

[0014] <First Embodiment> Referring to FIGS. 1 to 8, a first embodiment of the present invention will be described. Note that the imaging device 100 is described as having a function of cutting out (cropping) a partial image from a captured image and outputting the partial image to the outside. That is, an example in which the imaging device 100 also functions as a control device for the partial image will be described. However, a system configuration in which devices external to the imaging device 100, such as the client device 200 and the controller 400, have the function of a control device in each embodiment may also be used.

[0015] FIG. 1 is a diagram showing a system configuration in the present embodiment. The system in the present embodiment includes an imaging device (control device) 100, a client device 200, a network 300, a camera controller 400, and a display 500.

[0016] The imaging device 100 includes a video output terminal 101, a control communication terminal 102, and a network I / F 103. The video output terminal 101 is a terminal for outputting the video captured by the imaging device 100. As an example of the video output terminal, there are terminals conforming to standards such as SDI and HDMI (registered trademark), but the video may be packetized by IP (Internet Protocol) and output from an Ethernet (registered trademark) terminal.

[0017] The control communication terminal 102 is a communication terminal that receives and responds to a control signal or a control command from the camera controller 400. As an example of the control communication terminal, there are contact terminals that simply transmit High / Low signals, serial communication standards such as RS-232C, and communication standards such as Ethernet (registered trademark). The network I / F 103 is a terminal for communicating with the client device 200 via the network 300. As an example of the network I / F 103, there are Ethernet (registered trademark) terminals and Wi-Fi (registered trademark) transceiver modules.

[0018] The client device 200 is an information processing device such as a personal computer equipped with a display unit, for example. The client device 200 displays the video and UI (User Interface) generated by the imaging device 100 based on the information received from the imaging device 100 via the network 300. Further, the client device 200 receives user operations via the displayed UI, and transmits various control instructions to the imaging device 100 via the network 300 based on the operations.

[0019] The imaging device 100 and the client device 200 are interconnected via the network 300. The network 300 is realized by a plurality of routers, switches, cables, etc. compliant with a communication standard such as Ethernet (registered trademark), for example. Note that the network 300 may be realized by the Internet, a wired LAN (Local Area Network), or the like.

[0020] The camera controller 400 is a controller equipped with a stick and control buttons. When the stick is tilted to the left, an instruction for panning movement in the left direction is sent as a control command via the control communication terminal 102 of the imaging device 100. When the stick is tilted upward, an instruction for tilting movement in the upward direction is sent as a control command via the control communication terminal 102 of the imaging device 100. At this time, the pan and tilt movement speeds are determined according to the angle at which the stick is tilted, and a control command specifying the movement speed is sent simultaneously. Also, by twisting the stick, the zoom of the imaging device 100 can be changed, and a control command instructing zoom-in when the stick is twisted clockwise and zoom-out when the stick is twisted counterclockwise is sent to the imaging device 100. The stick may be used to change the position and size of the crop range (the range of the partial image) described later. For example, when the stick is tilted to the left, an instruction to move the position of the crop range to the left is sent to the imaging device 100. When the stick is tilted upward, an instruction to move the position of the crop range upward is sent to the imaging device 100. When the stick is twisted clockwise, an instruction to reduce the crop range is sent to the imaging device 100. When the stick is twisted counterclockwise, an instruction to increase the crop range is sent to the imaging device 100. The control buttons are used for changing the internal settings of the camera controller 400 and for controlling the camera 100 other than PTZ. For example, it can be used for selecting which of the multiple crops described later is to be output to the video output terminal 101. It is also used for setting whether to control the PTZ of the camera 100 or to change the position and size of the crop range by the operation of the stick described above.

[0021] The display 500 receives the output video from the camera and displays it to the user.

[0022] Referring to FIGS. 2 and 3, the imaging device 100 according to the present embodiment will be described. FIG. 2 is an example of an external view of the imaging device 100 according to the present embodiment, FIG. 2(a) is a view seen from above, and FIG. 2(b) is a view seen from the side. FIG. 3 is an example of a functional block of the imaging device 100 according to the present embodiment.

[0023] The direction in which the optical axis of the lens 104 points is the imaging direction of the imaging device 100, and the light beam that has passed through the lens 104 forms an image on the imaging element of the imaging unit 110 of the imaging device 100. The lens driving unit 105 is composed of a driving system that drives the lens 104 and changes the focal length of the lens 104. The lens driving unit 105 is controlled by the pan-tilt-zoom control unit 113.

[0024] The pan driving unit 106 is composed of a mechanical driving system that performs a pan operation and a motor as a driving source, and drives to perform control for rotational driving to rotate the imaging direction of the imaging device 100 in the pan direction 108. The pan driving unit 106 is controlled by the pan-tilt-zoom control unit 113.

[0025] The tilt driving unit 107 is composed of a mechanical driving that performs a tilt operation and a motor as a driving source, and drives to perform control for rotational driving to rotate the imaging direction of the imaging device 100 in the tilt direction 109. The tilt driving unit 107 is controlled by the pan-tilt-zoom control unit 113.

[0026] The imaging unit 110 is composed of an imaging element (not shown) such as a CCD (charge coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor. Then, the imaging unit 110 photoelectrically converts the subject image formed through the lens 104 to generate an electrical signal. The image processing unit 111 performs image processing such as processing for converting the electrical signal photoelectrically converted in the imaging unit 110 into a digital signal and compression encoding processing, and generates image data. Also, based on an instruction from the system control unit 112, the image processing unit 111 cuts out pixels in a predetermined range from the image, performs enlargement or reduction processing, and generates a partial image.

[0027] Based on the instruction transmitted from the system control unit 112, the pan-tilt-zoom control unit 113 controls the pan, tilt, and zoom of the imaging device 100 by controlling the pan driving unit 106, the tilt driving unit 107, and the lens driving 106.

[0028] The memory unit 114 stores (holds) setting values such as, for example, image quality setting parameters, the position of pan-tilt-zoom, the position and size of the crop range, etc.

[0029] The video output unit 115 converts the video generated by the image processing unit 111 into a predetermined video format and outputs it via the video output terminal 101. Also, when the output video is a crop video and is a vertical crop described later, the video output unit 115 rotates the video by 90° and outputs it.

[0030] The control communication unit 116 transmits the received data received from the camera controller 400 via the control communication terminal 102 to the system control unit 112 and performs camera control such as pan-tilt-zoom.

[0031] The network communication unit 117 communicates with the client device 200 via the network I / F 103. For example, the network communication unit 117 transmits the image data of the image captured by the imaging device 100 to the client device 200 via the network 300. Also, the network communication unit 117 receives a control command, which is a command for controlling the imaging device 100 transmitted from the client device 200, and transmits it to the system control unit 112.

[0032] The image analysis unit 118 performs image analysis on the image generated by the image processing unit 111 based on an instruction from the system control unit 112 and transmits it to the system control unit 112. Examples of image analysis include human body detection, face detection, moving object detection, etc. For example, it detects a human body in the captured image and transmits its position to the system control unit 112. However, depending on the use scene, face detection or moving object detection may be used instead.

[0033] The system control unit 112 is equipped with a CPU (Central Processing Unit) and controls the entire imaging device 100 through processes executed by the CPU, performing, for example, the following processes. That is, the system control unit 112 analyzes a control command for controlling the imaging device 100 transmitted from the client device 200 and performs a process corresponding to the control command. Also, the system control unit 112 gives an instruction for a pan-tilt-zoom operation to the pan-tilt-zoom control unit 113. Further, the system control unit 112 gives instructions such as setting the image quality adjustment and generating a partial image when the image processing unit 111 generates image data. Also, the system control unit 112 performs, for example, the following processes according to the image analysis result from the image analysis unit 118. That is, the system control unit 112 receives the position of the human body detected by the image analysis unit 118, determines the imaging direction of the imaging device 100 according to the position, and issues a pan-tilt-zoom drive instruction to the pan-tilt-zoom control unit 113 to perform human body tracking processing. Or, the system control unit 112 receives the position of the human body detected by the image analysis unit 118, determines the position and size of the crop range according to the position, and issues a crop instruction to the image processing unit 111 to move the crop range so as to track the human body.

[0034] Note that the imaging range in this embodiment is determined by the pan value, tilt value, and zoom value (in other words, the PTZ position) of the imaging device 100, and the pan value and tilt value among them are expressed by the coordinate system shown in FIG. 2.

[0035] As shown in FIG. 2(a), the pan value is the angle of the imaging direction (optical axis) in the pan direction 108 of the imaging device 100 when the center of the two drive ends of the pan drive unit 106 is set to 0°. The pan value can take a value in the range of -180° to +180° with clockwise being positive when the imaging device 100 is viewed from above.

[0036] The tilt value is the angle of the imaging direction (optical axis) in the tilt direction 109 of the imaging device 100 when the imaging direction of the imaging device 100 is directed in a direction parallel to the installation surface of the imaging device 100 as shown in FIG. 2(b), with the tilt value being set to 0°. The tilt value can take a value in the range of -40° to +220°, with the direction upward from the 0° position being defined as positive.

[0037] Note that the zoom value of the imaging device 100 when an image is captured by the imaging push value 100 is calculated from the focal length of the lens 104.

[0038] Referring to FIG. 4, a coordinate system for expressing the position and size of the crop range (partial image) in the crop function (cutting out of a partial image) according to the present embodiment will be described.

[0039] The captured image 600 corresponds to the entire image captured by the imaging device 100, and its angle of view is determined by the PTZ position of the imaging device 100. In the present embodiment, the aspect ratio of the captured image 600 is 16:9, and the number of pixels is 3840 pixels in width and 2160 pixels in height. The coordinate system defined on the captured image 600 has the upper left end as the origin, with the right direction being the X direction and the lower direction being the Y direction. That is, when the coordinates of a pixel on the captured image 600 are (X, Y), the coordinates can range from (0, 0) to (3840, 2160).

[0040] The crop ranges shown in the figure are, as an example, two crop ranges: crop range 1 which is 601 and crop range 2 which is 602. The aspect ratios of the crop ranges are, as an example, a horizontal crop of 16:9 and a vertical crop of 9:16. In the example shown in the figure, it is assumed that crop range 601 is a vertical crop and crop range 602 is a horizontal crop.

[0041] The position of the crop range is represented by the position of the center of the crop range. For example, the position of crop range 601 is represented as (X1, Y1). As an example, it is represented in a coordinate system with the upper left end of the captured image as the origin. That is, X1 is the X coordinate in such a coordinate system, and Y1 is the Y coordinate.

[0042] The size of the crop range is represented by the number of pixels in the vertical and horizontal directions. For example, the width of crop range 601 is represented as W1, and the height is represented as H1.

[0043] In the following description, the position and size of the nth crop range n are collectively represented as (Xn, Yn, Wn, Hn). That is, the position and size of 601, which is crop range 1, are (X1, Y1, W1, H1), and the position and size of 602, which is crop range 2, are (X2, Y2, W2, H2).

[0044] However, it is not necessary for the number of crop ranges to be two, and three or more are also possible. Also, the aspect ratio of the crop range is not limited to a predetermined ratio and may be any aspect ratio.

[0045] Referring to FIG. 5, a UI example for setting the crop according to this embodiment will be described. This UI is displayed on the display of the client device 200 based on the communication content via the network 300 performed between the imaging device 100 and the client device 200. The user can obtain information about the camera 100 through this UI and perform settings related to the crop function via an operation device such as a mouse or a touch panel. Based on the user operation performed on this UI, the client device 200 transmits a setting command to the imaging device 100 via the network 300.

[0046] The UI screen 700 is a screen of the UI related to the crop. The UI screen 700 is displayed, for example, as a screen of a web browser or a window of a dedicated application. All the parts of the UI described below are displayed on the UI screen 700.

[0047] On the UI screen 700, the captured image 600 is displayed. Also, on the captured image 600, the crop range 601 and the crop range 602 are superimposed and displayed, and the user can visually confirm the image captured by the current imaging device 100 and the position and size of each crop range. Also, the user can use a mouse cursor or the like to perform operations such as moving the position and scaling the size of the crop range by dragging and dropping the crop range 601 or the crop range 602.

[0048] The pan operation slider bar 701 and the pan operation button 702 are UI parts for the user to operate the pan of the imaging device 100. By moving the pan operation slider bar 701 by dragging the mouse or the like, an arbitrary pan position can be specified. The pan operation button 702 consists of a left button and a right button, and while the left button is pressed, a command to drive the pan to the left and while the right button is pressed, a command to drive the pan to the right are sent to the imaging device 100.

[0049] The tilt operation slider bar 703 and the tilt operation button 704 are UI parts for the user to operate the tilt of the imaging device 100. By moving the tilt operation slider bar 703 by dragging the mouse or the like, an arbitrary tilt position can be specified. The tilt operation button 704 consists of an up button and a down button, and while the up button is pressed, a command to drive the tilt up and while the down button is pressed, a command to drive the tilt down are sent to the imaging device 100.

[0050] The zoom operation slider bar 705 and the zoom operation button 706 are UI parts for the user to operate the zoom of the imaging device 100. By moving the zoom operation slider bar 705 by dragging the mouse or the like, an arbitrary zoom position can be specified. The zoom operation button 706 is composed of a Tele button and a Wide button. While the Tele button is pressed, a command to drive the zoom in the tele direction (the direction of zooming in) and while the Wide button is pressed, a command to drive the zoom in the wide direction (the direction of zooming out) are sent to the imaging device 100.

[0051] The crop operation block 710 is a block that gathers UI parts for the user to set the position and size of the crop range, and includes the UI parts described below.

[0052] The operation target crop selection box 711 sets the crop range to be operated on in the crop operation block 710. In this embodiment, "Crop 1" and "Crop 2" can be selected in this selection box. When "Crop 1" is selected, the crop range 601, and when "Crop 2" is selected, the crop range 602, become the targets to be operated on by the UI parts included in the crop operation block 710.

[0053] The cross button 712 for crop movement is a button for operating the position of the crop range. The cross button 712 for crop movement is composed of a left button, a right button, an up button, and a down button. While the left button is pressed, a command to move the crop range in the left direction is sent to the imaging device 100. While the right button is pressed, a command to move the crop range in the right direction is sent. While the up button is pressed, a command to move the crop range in the up direction is sent. While the down button is pressed, a command to move the crop range in the down direction is sent.

[0054] The crop zoom operation slider bar 713 and the crop zoom operation button 714 are UI parts for the user to operate the size of the crop range. Here, "crop zoom" corresponds to the size of the crop range. When the crop range is reduced, the range is enlarged for display, and when the crop range is enlarged, the range is reduced for display. Therefore, here, reducing the crop range is expressed as moving the crop zoom in the tele direction, and enlarging the crop range is expressed as moving the crop zoom in the wide direction. By moving the crop zoom operation slider bar 713 by dragging the mouse or the like, the size of an arbitrary crop range can be specified. The upward direction of the crop zoom operation slider bar 713 is the tele direction of the crop zoom, and the downward direction is the wide direction of the crop zoom. The crop zoom operation button 714 consists of a Tele button and a Wide button. While the Tele button is pressed, a command to move the crop zoom in the tele direction is sent to the imaging device 100, and while the Wide button is pressed, a command to move the crop zoom in the wide direction is sent to the imaging device 100.

[0055] The crop movement speed operation slider bar 715 and the crop movement speed operation button 716 are UI parts for specifying the speed of movement of the crop range and the crop zoom. The movement of the crop range and the movement of the crop zoom while the cross button 712 for crop movement and the crop zoom operation button 714 are pressed are performed at a speed corresponding to the crop movement speed. By moving the crop movement speed operation slider bar 715 by dragging the mouse or the like, the movement speed of the crop can be specified. The crop movement speed operation button 716 consists of a Fast button and a Slow button. While the Fast button is pressed, a command to increase the crop movement speed is sent to the imaging device 100, and while the Slow button is pressed, a command to decrease the crop movement speed is sent to the imaging device 100.

[0056] The crop output setting block 720 is a block that selects which cropped video to output or whether to output the full-screen video instead of the cropped video for each user video output, and is provided with an output selection box 721 for each video output. In this embodiment, the video output systems include SDI, Stream 1, Stream 2, and Stream 3. SDI is the video output via the video output terminal 101, and Streams 1 to 3 are the videos output as network streams via the network I / F 103. In the example of FIG. 5, the setting is to output Crop 1 video to SDI, full-screen video to Streams 1 and 3, and Crop 2 video to Stream 2.

[0057] The crop angle fixing block 730 is a block for setting whether to fix the angle of view of the crop, and is provided with an angle-of-view fixing selection box 731 for each crop. The angle-of-view fixing setting has options of "Always OFF", "When subject is detected", "When outputting", and "Always ON". This is a setting regarding the determination method of whether to correct the position and size of the crop range when the PTZ position of the imaging device 100 is moved. The correction of the crop range during PTZ position movement will be described in detail later. When "Always OFF" is selected, it is not the target of crop range correction. When "When subject is detected" is selected, it is the target of crop range correction when a target subject (e.g., a person) is detected within the crop range. When "When outputting" is selected, it is the target of crop range correction when the target crop is output to any of the video output systems of SDI or Streams 1 to 3. When "Always ON" is selected, it is always the target of crop range correction. Whether it is set to "Always ON" or "Always OFF" is managed by a flag. The storage unit 114 stores the setting value of this flag.

[0058] The angle-of-view priority setting block 740 is a block for setting which angle of view, the full-screen or the crop, is to be prioritized, and is provided with an angle-of-view priority setting radio button 741. When the full screen is selected with the angle-of-view priority setting radio button 741, when the position or size of the crop range is changed, the process of moving the PTZ accordingly is not performed. When the crop is selected with the angle-of-view priority setting radio button 741, PTZ movement may be performed when the position or size of the crop range is changed. Also, when performing PTZ movement, a constraint is imposed such that it cannot be moved to a PTZ position where the angle of view of the crop cannot be maintained.

[0059] The crop auto-tracking setting block 750 is a block for setting the auto-tracking of the crop, and is provided with a crop auto-tracking switch 751 for each crop. For a crop in which the crop auto-tracking switch 751 is ON, the crop range is moved so as to automatically track the subject within the crop range. Detection of the subject within the crop range is performed by the image analysis unit 118, and when a subject exists within the crop range, the position and size of the crop range are automatically changed so as to track the subject.

[0060] With reference to FIGS. 6, 7, and 8, the correction process when changing the angle of view of the crop according to this embodiment will be described. FIG. 6 is a flowchart of the process when changing the crop range of the imaging device 100. FIG. 7 is a flowchart for explaining in detail the process of step S602 in FIG. 6. FIG. 8 is a diagram showing the change in the output of the captured image 600 and each partial image. The partial image 801 of the crop range 1 is a video obtained by cutting out the crop range 601 and outputting it. The partial image 802 of the crop range 2 is a video obtained by cutting out the crop range 602 and outputting it. Here, the description will be given by taking as an example the change of increasing the size of the crop range 601, but the same applies to the case of the change with respect to the crop range 602.

[0061] In step S601, the imaging device 100 acquires, via the network communication unit 117, the details of the position and size change for the crop range 601. As an example of this change detail, it is a command to change the size of the crop range 601 transmitted from the client device 200 based on a user operation using the crop zoom operation slider bar 713 or the crop zoom operation button 714. In this case, the command sent to the system control unit 112 via the network communication unit 117 is analyzed to acquire the details of the position and size change of the crop range 601. Also, in addition to this, when the automatic tracking of the crop for the crop range 601 is enabled, it may be a movement or a change in size of the crop range based on the automatic tracking result. In this case, the system control unit 112 determines the position and size of the crop range 601 based on the position of the tracking target detected by the image analysis unit 118. Fig. 8(a) shows the captured image and the state of each crop range at this point. Here, it is assumed that a command to enlarge the crop range 601, as indicated by the dashed line in the figure, is received.

[0062] In step S602, based on the acquired details of the position and size change of the crop range 601 and the current settings, the system control unit 112 determines the PTZ position and the position and size of the crop range 601. The determination method will be described in detail later with reference to Fig. 7.

[0063] In step S603, based on the determination in step S602, the system control unit 112 changes the position and size of the crop range 601.

[0064] In step S604, based on the determination in step S602, the system control unit 112 determines whether it is necessary to move the PTZ position of the imaging device 100. If PTZ movement is not required, the process ends here. If PTZ movement is necessary, the process proceeds to step S605. As described above, when the process of changing the position and size of the first partial image cut out from the captured image is performed, the system control unit 112 functions as a determination means for determining whether to change the angle of view of the captured image.

[0065] In step S605, the system control unit 112 starts moving the angle of view of the captured image to the PTZ position determined in step S602. Thereafter, steps S606 to S610 are repeatedly executed until the PTZ position reaches the target position. In this way, when it is determined that the angle of view of the captured image is to be changed, the system control unit 112 also functions as an angle-of-view control means for controlling the angle of view of the captured image so as to include the area of the first partial image after being changed by the change process.

[0066] In step S606, the current PTZ position is acquired. The current PTZ position can be acquired via the pan-tilt-zoom control unit 113.

[0067] In step S607, as the PTZ position changes, the system control unit 112 determines whether correction of the crop range 602 is necessary.

[0068] This determination may be made, for example, in consideration of the setting contents of the output selection box 721 and the angle-of-view fixed selection box 731. In that case, the following three conditions may be considered as the conditions for determining that correction of the crop range 602 is necessary.

[0069] The first is the case where in the angle-of-view fixed selection box 731, Crop 2 is set to "always ON". In this case, it is determined that correction of the crop range 602 is necessary. Whether it is set to "always ON" is determined by a flag.

[0070] The second is the case where in the angle-of-view fixed selection box 731, Crop 2 is set to "when subject is detected", and a specific target subject is detected within the crop range 2. In this case, it is determined that correction of the crop range 602 is necessary.

[0071] The third case is when the angle-of-view fixed selection box 731 has Crop 2 set to "at output", and "Crop 2" is set in any of the selection boxes of the output selection box 721. That is, when the partial image based on Crop 2 is output to the outside. In this case, it is determined that correction of the crop range 602 is necessary.

[0072] If none of the above conditions are met, it is considered that correction of the crop range 602 is unnecessary, and the process proceeds to step S610. Note that the above three conditions are just examples, and the crop range may always be corrected without making a determination based on these conditions. That is, the position and size of another crop range may always be corrected regardless of a flag or the like.

[0073] If it is determined that correction of the crop range 602 is necessary, the process proceeds to step S608.

[0074] In step S608, the system control unit 112 calculates the position and size of crop range 2 where the angle-of-view of the crop range 602 does not substantially change. That is, even when the angle-of-view of the captured image changes due to PTZ movement, the position and size of crop range 2 that results in a partial image of substantially the same subject are calculated. This calculation method will be described in detail below.

[0075] Here, the PTZ position is represented as (P, T, Z). Let the PTZ position before the start of PTZ movement, that is, at the time of step S605, be (Pa, Ta, Za), and the PTZ position at the time of execution of the current step S608 be (Pb, Tb, Zb). However, the zoom value Z is represented by the horizontal angle-of-view here, and the unit is [deg]. Also, let the position and size of crop range 2 before the start of PTZ movement, that is, at the time of step S605, be (X2a, Y2a, W2a, H2a), and the position and size of crop range 2 after correction be (X2b, Y2b, W2b, H2b). Then, since the number of pixels of the captured image 600 is 3840x2160, (X2b, Y2b, W2b, H2b) can be obtained by the following formula (1).

[0076]

Number

[0077] In step S609, the system control unit 112 controls the image processing unit 111 to change the position and size of the crop range 2 to the position and size calculated by equation (1). In this way, when the angle of view of the captured image is changed, the system control unit 112 also functions as a cropping control means for controlling to change the position and size of the crop range 2 with respect to the captured image after the change.

[0078] In step S610, it is determined whether the PTZ has reached the target position. If it has not reached, the process returns to step S606 again. If the PTZ has reached the target position, the process proceeds to step S611, the PTZ is stopped, and the flow ends.

[0079] Figure 8(b) is a diagram showing the captured image 600 after a series of flows are completed, each crop range, and the corresponding partial images. The angle of view of the partial image 801 of the crop range 1 is widely changed, and as a result, the angle of view (subject) of the partial image 802 of the crop range 2 has not changed.

[0080] The processing content of step S602 will be described in detail with reference to FIG. 7.

[0081] In step S701, the system control unit 112 determines whether the crop range 601 protrudes from the allowable range of the crop range within the captured image 600 at the changed position and size of the crop range 601 obtained in step S601. The allowable range of the crop range is the range within which the crop range must fit within the captured image 600. As a method for determining the allowable range of the crop range, for example, it can be determined as a range obtained by excluding an outer edge region of a certain number of pixels (assumed to be K pixels) from the angle-of-view end of the captured image 600. The position and size (X, Y, W, H) of the crop range need to satisfy the following equation (2).

[0082]

Number

[0083] When all the constraints of Equation (2) are satisfied, it can be determined that the crop range 1 is within the allowable range of the crop range. The reason for setting the allowable range of the crop range is that if the crop range is too close to the angle-of-view edge of the captured image 600, when the automatic crop tracking is turned on, even if the subject moves slightly, it will go out of the range of the captured image 600 and tracking will become impossible. Therefore, by providing the allowable range of the crop range, it is possible to prevent the automatic crop tracking from being interrupted.

[0084] In step S702, the system control unit 112 determines whether the crop range 601 of the change destination extends outside the allowable range of the crop range. If it does not extend, it proceeds to step S703; if it extends, it proceeds to step S704.

[0085] In step S703, since there is no need to change the PTZ, the system control unit 112 adopts the position and size of the crop range 1 of the change destination as they are and ends the flow.

[0086] In step S704, in order to determine whether to move the PTZ, the system control unit 112 checks the setting of angle-of-view priority. If it is crop range priority, since the PTZ position needs to be moved, it proceeds to step S705. If it is full-screen priority, since the PTZ position is not moved, it proceeds to step S709. Note that step S704 may be omitted and the process may proceed directly to step S705.

[0087] In step S705, the system control unit 112 performs a calculation to obtain a PTZ position such that all the crop ranges with fixed angles of view are within the imaging range. An example of the calculation method for obtaining this PTZ position is described below. Here, it is assumed that the crop range 2 has a fixed angle of view for explanation.

[0088] Let the position and size of the cropping range 601 to be changed be (X1a, Y1a, W1a, H1a). Also, let the position and size of the cropping range 602 at this time be (X2a, Y2a, W2a, H2a). Furthermore, let the number of pixels that can accommodate all the cropping ranges be Wcrop_a and Hcrop_a respectively. In this case, the condition for including the cropping range 1 and the cropping range 2 in the captured image only by driving the PT without changing the zoom is that the following formula (3) holds.

[0089]

Number

[0090] However, K is the number of pixels in the outer edge area for determining the cropping range allowable range as described above. When the inequality in formula (3) holds, there is no need to move the zoom in the wide direction. Therefore, the zoom can be moved to a PT position that can accommodate the cropping range 601 and the cropping range 602 while remaining fixed. When formula (3) does not hold, it is necessary to move the zoom in the wide direction, so the zoom position is calculated.

[0091] Let the horizontal viewing angle of the current zoom be Za and the horizontal viewing angle of the zoom after movement be Zb. Also, let the position and size of the cropping range 601 to be changed and the position and size of the current cropping range 602 after applying the correction for the zoom movement be (X1b, Y1b, W1b, H1b) and (X2b, Y2b, W2b, H2b) respectively. Here, the following formula (4) needs to hold for the position and size of the cropping range after correction.

[0092]

Number

[0093] Here, the positional relationship between the cropping range 601 and the cropping range 602 does not change depending on the zoom, and the correction is equally lacking for both. Therefore, the following formula (5) holds.

[0094]

Number

[0095] Therefore, if Wcrop_a and Hcrop_a are obtained, Zb that can satisfy both of the two equations shown in the following equation (6) may be set as the zoom position.

[0096]

Number

[0097] From the above, after calculating the zoom amount, it is only necessary to move to the PT position where both the crop range 601 and the crop range 602 can be included in the captured image at the changed zoom position.

[0098] In step S706, if there exists a Zb that satisfies both of the above equations (6), assuming that there exists a suitable PTZ position, the process proceeds to step S707. If the equations (6) cannot be satisfied even when the zoom is pulled to the wide end, assuming that there does not exist a suitable PTZ position, the process proceeds to step S708.

[0099] In step S707, the system control unit 112 sets the PTZ position obtained in step S705 as the determination position and directly proceeds to step S710.

[0100] In step S708, the system control unit 112 determines to set the zoom setting to the wide end, and determines the PT position to be the position where the crop range 601 can be maximized. The PT position can be determined according to the positional relationship between the crop range 601 and the crop range 602. For the horizontal direction, assume that the X position X1 of the crop range 601 is smaller than the X position X2 of the crop range 602. In this case, the right end of the crop range 2 is set to be the right end of the crop range allowable range. Also, when X1 is larger than X2, the pan position is determined so that the left end of the crop range 2 is the left end of the crop range allowable range. For the vertical direction, when the Y position Y1 of the crop range 601 is smaller than the Y position Y2 of the crop range 602, the lower end of the crop range 2 is set to be the lower end of the crop range allowable range. Also, when Y1 is larger than Y2, the tilt position is determined so that the upper end of the crop range 2 is the upper end of the crop range allowable range.

[0101] In step S709, the system control unit 112 determines the size of the crop range 1 to be the maximum size that can fit within the captured image. Since the PTZ position has already been determined, the maximum size that satisfies the above formula (2) with respect to the determined PTZ position and the position of the crop range 601 may be determined. By this procedure, as shown in FIG. 8(c), the viewing angle of the partial image 801 of the crop range 1 can be maximized without changing the viewing angle of the partial image 802 of the crop range 2.

[0102] In step S710, the system control unit 112 corrects the position and size of the crop range 601 as it moves to the PTZ position obtained in the previous steps. The correction at this time can be performed by using the above formula (1).

[0103] The above is the first embodiment. By controlling the cutout range as described above, even when a plurality of partial images are cut out from the captured image, the range of the cutout image can be appropriately changed.

[0104] <Second Embodiment> Next, with reference to FIGS. 9 to 11, a second embodiment of the present invention will be described.

[0105] Note that the system configuration diagram, the external view of the imaging device, the functional block diagram, and the coordinate system of the crop range according to this embodiment are the same as those described with reference to FIGS. 1 to 5 in the first embodiment, and thus the description will be omitted as appropriate. In addition, the description of the same parts as those in the first embodiment will be omitted as appropriate.

[0106] In the first embodiment, the imaging device 100 acquires the change content of the position and size of the crop range, and determines the change content of the PTZ position and the correction content of the position and size of the crop range. However, the determination of the change content of the PTZ position and the correction content of the position and size of the crop range may be performed on the client device 200 side. In this embodiment, an example will be described in which the determination of the change content of the PTZ position and the correction content of the position and size of the crop range is performed on the client device 200 side and transmitted to the imaging device 100 as a command via the network 300.

[0107] FIG. 9 is a flowchart of the processing on the client device 200 side when changing the position and size of the crop range 601 according to this embodiment, and FIG. 10 is an example of a communication sequence between the imaging device 100 and the client device 200. A series of communications is realized by requests and responses of HTTP (Hypertext Transfer Protocol) on a communication path using, for example, a wired LAN. The data exchanged here will be described later. In this embodiment, it is assumed that functions equivalent to the system control unit 112 and the network communication unit 117 are inside the client device 200, and the client device 200 acquires necessary information by communicating with the imaging device 100.

[0108] In step S901, the system control unit 112 acquires the change content of the position and size of the crop range 601. This change content is, for example, the operation content by the crop movement cross button 712 or the crop zoom operation slider bar 713.

[0109] In step S902, the system control unit 112 determines the PTZ position and the position and size of the crop range 601. Since this determination method is the same as the content of step S602 described with reference to FIG. 6 of the first embodiment, the description thereof is omitted.

[0110] In step S903, the system control unit 112 transmits a command specifying the position and size of the crop range 601 to the imaging device 100.

[0111] In step S904, the system control unit 112 determines whether PTZ movement is necessary based on the determination content of step S902. If PTZ movement is not necessary, this flow ends as it is. If PTZ movement is necessary, the process proceeds to step S905.

[0112] In step S905, the system control unit 112 transmits a command instructing PTZ movement to the imaging device 100.

[0113] In step S906, the system control unit 112 acquires the current PTZ position of the imaging device 100. The acquisition of the PTZ position is realized by transmitting a PTZ position information request command from the client device 200 to the imaging device 100, and the imaging device 100 transmits the PTZ position information as a response thereto.

[0114] In step S907, the system control unit 112 determines whether correction of the position and size of the crop range 602 is necessary. Since this determination method is the same as the process of step S607 described with reference to FIG. 6 in the first embodiment, the description thereof is omitted.

[0115] In step S908, the system control unit 112 calculates the position and size such that the viewing angle of the crop range 602 does not change. Since this process is the same as the process of step S608 described with reference to FIG. 6 in the first embodiment, the description thereof is omitted.

[0116] In step S909, the system control unit 112 causes the network communication unit 117 to send a command to the apparatus 100 to change the position and size of the crop range 2.

[0117] In step S910, it is determined whether or not the PTZ has reached the target position. If it has not reached the target position, the process returns to step S906 again. If it has reached the target position, this flow ends.

[0118] FIG. 11 is an example of the packet structure of data exchanged in the communication sequence described with reference to FIGS. 9 and 10. FIG. 11(a) is an example of transmission data from the client apparatus 200 to the imaging apparatus 100, and FIG. 11(b) is an example of transmission data from the imaging apparatus 100 to the client apparatus 200.

[0119] The transmission data from the client apparatus 200 to the imaging apparatus 100 consists of a header section, a command section, and a parameter section. The header section contains information such as the address representing the source device of the packet, the address representing the destination device of the packet, and the size of the packet. The command section represents the function of the imaging apparatus 100 controlled by this packet, and the parameter section represents the detailed control content of the command. The number of parameters in the parameter section varies depending on the content of the command. As the command, “PTZ_GET_POS” for requesting information on the current PTZ position of the imaging apparatus 100 is available. Also, “PTZ_SET_POS” for requesting PTZ movement by specifying the PTZ position of the imaging apparatus 100 and “CROP_GET_POS_SIZE” for requesting information on the position and size of an arbitrary crop range are also used. Furthermore, “CROP_SET_POS_SIZE” for changing the position and size of an arbitrary crop range is also available. Note that in this embodiment, since the aspect ratio of the crop range is fixed, only the width of the crop range is specified in “CROP_SET_POS_SIZE”.

[0120] The transmission data from the imaging device 100 to the client device 200 consists of a combination of a header part, a plurality of key parts, and a data part. The header part contains information such as the address representing the source device of the packet, the address representing the destination device of the packet, and the size of the packet. The key part is an identifier indicating the content represented by the data, and the data part contains a value. As keys, "PAN", "TILT", and "ZOOM" indicating the PTZ position of the imaging device 100 can be used. Also, "CROP1_X", "CROP1_Y", "CROP1_W", "CROP1_H", "CROP2_X", "CROP2_Y", "CROP2_W", and "CROP2_H" indicating the position and size of each crop range are also used.

[0121] <The Third Embodiment> Referring to FIG. 12, the third embodiment of the present invention will be described.

[0122] Note that the system configuration diagram, the external view of the imaging device, the functional block diagram, the coordinate system of the crop range, and the processing flow when the crop range 601 is changed according to this embodiment are the same as those described with reference to FIGS. 1 to 7 in the first embodiment, so the description will be omitted. In addition, the description of parts similar to those in the first embodiment will be omitted as appropriate.

[0123] In the first embodiment, when the PTZ position changes along with the change of the crop range 601, the angle of view of the full-screen video changes. In this embodiment, when the PTZ position is changed, if the imaging area after the change includes the imaging area before the change, a process is included in which the cut-out video is output as the full-screen video so that the angle of view of the full-screen video does not change.

[0124] FIG. 12 is a diagram showing the imaging image and each crop range in this embodiment, and the changes in the full-screen video and each crop video. FIG. 12(a) represents before the change of the crop range 601, and FIG. 12(b) represents after the change. The full-screen video 800 is a full-screen video without cropping, and is output to the output system in which "full screen" is specified in the output selection box 721.

[0125] As shown by the dashed line in Fig. 12(a), when the size of the crop range 601 is changed, the crop range 601 goes out of the range of the captured image 600. At this time, the PTZ position of the imaging device 100 is changed according to the flow described with reference to Figs. 6 and 7 of the first embodiment. Furthermore, by correcting the position and size of the crop range 602, the viewing angle of the partial image 801 of the crop range 1 can be widened without changing the viewing angle of the partial image 802 of the crop range 2. At this time, the system control unit 112 sets a full-screen video cutout range 620 within the captured image 600. The position and size of the full-screen video cutout range 620 can be determined by performing the calculation process implemented in step S608 of Fig. 6 on the entire range of the captured image 600 in the same manner as the crop range 602. The system control unit 112 causes the image processing unit 111 to perform cutout and resizing processes similar to cropping according to the determined full-screen video cutout range 620, thereby generating the full-screen video 800. As described above, as shown in Fig. 12(b), the viewing angle of the partial image 801 of the crop range 1 can be widened without changing the viewing angle of the full-screen video 800 before and after the change of the PTZ position.

[0126] (Other embodiments) Next, with reference to Fig. 13, the hardware configuration for realizing each function of the control device of each embodiment will be described. Note that each configuration of the camera other than the imaging element and the lens in each embodiment may also be realized by the hardware configuration shown in Fig. 13. These hardware components are included in the imaging device 100, the client device 200, or the controller 400.

[0127] The RAM (Random Access Memory) 1302 temporarily stores the computer programs executed by the CPU (Central Processing Unit) 1301. Also, the RAM 1302 temporarily stores data (such as commands and video data) acquired from the outside via the communication interface 1304. Further, the RAM 1302 provides a work area used when the CPU 1301 executes various processes. Also, the RAM 1302 functions as, for example, a frame memory or a buffer memory.

[0128] The CPU 1301 executes the computer programs stored in the RAM 1302. In addition to the CPU, other processors such as DSP (Digital Signal Processor) or ASIC (Application Specific Integrated Circuit) may be used.

[0129] The HDD (Hard Disk Drive) 1303 stores the operating system programs and video data. Also, the HDD 1303 stores computer programs.

[0130] The computer programs and data stored in the HDD 1303 are appropriately loaded into the RAM 1302 according to the control by the CPU 1301 and executed by the CPU 1301. In addition to the HDD, other storage media such as flash memory may be used. The bus 1305 connects each hardware. Each hardware exchanges data via the bus 1305. The above is the hardware configuration in each embodiment.

[0131] Note that the present invention can also be realized by a process in which one or more processors read and execute a program that realizes one or more functions of the above-described embodiments. The program may be supplied to a system or device having a processor via a network or a storage medium. Also, the present invention can be realized by a circuit (for example, an ASIC) that realizes one or more functions of the above-described embodiments.

[0132] In addition, the control device in each embodiment may be realized by the hardware shown in FIG. 13, or may be realized by software.

[0133] Moreover, the present invention is not limited to each of the embodiments described above, and various modifications can be made without departing from the gist of the present invention. For example, combinations of each embodiment and modification example are also included in the disclosure of this specification.

[0134] The disclosure of this specification includes the following image processing apparatus, image processing method, and program.

[0135] (Item 1) When a process of changing at least one of the position and size of a first partial image cut out from a captured image captured by an imaging unit is performed, determination means for determining whether to change the angle of view of the captured image; When it is determined by the determination means that the angle of view of the captured image is to be changed, angle-of-view control means for performing control to change the angle of view of the captured image so as to include the region of the first partial image after being changed by the process; When the angle of view of the captured image is changed by the angle-of-view control means, Cut-out control means for performing control to change at least one of the position and size with respect to the changed angle of view of the captured image in a second partial image cut out from the captured image and different from the first partial image; A control device characterized by comprising:

[0136] (Item 2) The determination means determines whether to change the angle of view of the captured image based on the position of the first partial image and the content of the process. The control device according to Item 1, characterized in that:

[0137] (Item 3) The process is performed according to an operation by a user. The control device according to item 1 or 2, characterized in that...

[0138] (Item 4) The processing is performed according to the tracking result of the subject in the captured image. The control device according to any one of items 1 to 3, characterized in that...

[0139] (Item 5) The determination of whether to change the angle of view of the captured image is made based on a setting indicating which of the captured image set by the user and the first partial image is to be prioritized. The control device according to any one of items 1 to 4, characterized in that...

[0140] (Item 6) For each of the plurality of partial images, a flag indicating whether to correct at least one of the position and size in the captured image is stored. When the angle of view of the captured image is changed by the angle-of-view control means, the cutout control means determines whether to change at least one of the position and size of the second partial image in the captured image based on the flag in the second partial image. The control device according to any one of items 1 to 5, characterized in that...

[0141] (Item 7) When the angle of view of the captured image is changed by the angle-of-view control means, the cutout control means determines whether to change at least one of the position and size of the second partial image in the captured image based on whether the second partial image is being output externally. The control device according to any one of items 1 to 6, characterized in that...

[0142] (Item 8) When the angle of view of the captured image is changed by the angle-of-view control means, the cutout control means determines whether to change at least one of the position and size of the second partial image in the captured image based on whether a specific subject is included in the second partial image. The control device according to any one of Items 1 to 7, characterized in that...

[0143] (Item 9) When a process of changing at least one of the position and size of a first partial image cut out from a captured image captured by an imaging unit is performed, a determination step of determining whether to change the angle of view of the captured image; When it is determined in the determination step that the angle of view of the captured image is to be changed, an angle-of-view control step of performing control to change the angle of view of the captured image so as to include the region of the first partial image after being changed by the process; When the angle of view of the captured image is changed by the angle-of-view control step, A cutout control step of performing control to change at least one of the position and size with respect to the changed angle of view of the captured image in a second partial image cut out from the captured image and different from the first partial image; A control method characterized by comprising the above.

[0144] (Item 10) A program for causing a computer to execute the control method according to Item 9.

Explanation of Signs

[0145] 100 Imaging device 200 Client device 300 Network 400 Camera controller 111 Image processing unit 112 System control unit 113 Pan-tilt-zoom control unit 117 Network communication unit

Claims

1. When a process of changing at least one of the position and size of a first partial image cut out from a captured image captured by an imaging unit is performed, determination means for determining whether to change the angle of view of the captured image; When it is determined by the determination means that the angle of view of the captured image is to be changed, angle-of-view control means for performing control to change the angle of view of the captured image so as to include the region of the first partial image after being changed by the process; When the angle of view of the captured image is changed by the angle-of-view control means, Cut-out control means for performing control to change at least one of the position and size of a second partial image cut out from the captured image with respect to the angle of view of the captured image after the change, the second partial image being different from the first partial image A control device characterized by comprising:

2. The determination means determines whether to change the angle of view of the captured image based on the position of the first partial image and the content of the process. The control device according to claim 1, characterized in that:

3. The process is performed according to an operation by a user. The control device according to claim 1, characterized in that:

4. The process is performed according to a tracking result of a subject in the captured image. The control device according to claim 1, characterized in that:

5. The determination as to whether to change the angle of view of the captured image is made based on a setting indicating which of the captured image and the first partial image set by the user is to be prioritized. The control device according to claim 1, characterized in that:

6. A flag indicating whether to correct at least one of the position and size in the captured image is stored for each of a plurality of partial images, When the angle of view of the captured image is changed by the angle-of-view control means, the cut-out control means determines whether to change at least one of the position and size of the second partial image in the captured image based on the flag in the second partial image. The control device according to claim 1, characterized in that:

7. When the angle of view of the captured image is changed by the angle-of-view control means, the cut-out control means determines whether to change at least one of the position and size of the second partial image in the captured image based on whether the second partial image is output to the outside. The control device according to claim 1, characterized in that:

8. When the angle-of-view control means changes the angle of view of the captured image, the cropping control means determines whether to change at least one of the position and size of the second partial image in the captured image based on whether a specific subject is included in the second partial image. The control device according to claim 1, characterized in that.

9. When a process of changing at least one of the position and size of a first partial image cropped from a captured image captured by an imaging means is performed, a determination step of determining whether to change the angle of view of the captured image; When it is determined in the determination step that the angle of view of the captured image is to be changed, an angle-of-view control step of performing control to change the angle of view of the captured image so as to include the area of the first partial image after being changed by the process; When the angle of view of the captured image is changed by the angle-of-view control step, A cropping control step of performing control to change at least one of the position and size with respect to the changed angle of view of the captured image in a second partial image cropped from the captured image and different from the first partial image; A control method characterized by comprising.

10. A program characterized by causing a computer to execute the control method according to claim 9.

Images