Image processing device and method, and image processing system

Abstract

The present technology relates to an image processing device and method and an image processing system that make it possible to suppress the occurrence of a feeling of discomfort relating to a delay during camera operation. This image processing device comprises an image processing unit that, on the basis of an operation control signal indicating an operation quantity for changing the imaging range of a camera, and a change quantity signal indicating the change quantity by which the imaging range has changed, performs cut-out processing to cut out a section of an image that was imaged by the camera. The present technology can be applied to an image processing system.

Description

Image processing device and method, and image processing system

[0001] The present technology relates to an image processing device and method, and an image processing system, and more particularly to an image processing device and method, and an image processing system that can suppress the occurrence of discomfort due to delays when operating a camera.

[0002] Conventionally, many technologies related to video production using cameras have been proposed.

[0003] For example, one such technique has been proposed in which, when transmitting a captured image, a portion of the image is cut out and the cut-out image is transmitted (see, for example, Patent Documents 1 and 2).

[0004] JP 2014-3455 A JP 8-237590 A

[0005] Currently, video production using cameras is undergoing a transition from a system configuration based on SDI (Serial Digital Interface) to a system configuration based on IP (Internet Protocol), and remote production is becoming more common.

[0006] Previously, PTZ (Pan Tilt Zoom) camera operators operated the PTZ camera on location via a wired or dedicated line connection between the controller and the PTZ camera, which meant there was only a small delay before the operation was reflected, allowing for seamless operation.

[0007] Meanwhile, in recent years, remote production has seen an increase in the number of cases where only the PTZ camera itself is deployed on location, with the operator operating the PTZ camera from a remote location. This means that PTZ cameras are increasingly being operated in environments where there can be significant and uncontrollable delays between the PTZ camera and the controller.

[0008] In such an environment, there is a delay between the time the operator's operation is transmitted to the PTZ camera and the time the image (video) reflecting that operation is presented to the operator, and if the delay is large, the operator will feel uncomfortable operating the camera and will have difficulty operating it properly.

[0009] The present technology has been made in consideration of such circumstances, and is intended to suppress the occurrence of discomfort due to delays when operating a camera.

[0010] The image processing device according to the first aspect of the present technology includes an image processing unit that performs a cutout process to cut out a part of an image captured by the camera based on an operation control signal that indicates an operation amount for changing the shooting range of the camera and a change amount signal that indicates an amount of change by which the shooting range has changed.

[0011] The image processing method according to the first aspect of the present technology includes a step of performing a cutout process to cut out a part of an image captured by the camera based on an operation control signal indicating an operation amount for changing the shooting range of the camera and a change amount signal indicating an amount of change by which the shooting range has changed.

[0012] In a first aspect of the present technology, a cutout process is performed to cut out a part of an image captured by the camera based on an operation control signal indicating an operation amount for changing the shooting range of the camera and a change amount signal indicating an amount of change by which the shooting range has changed.

[0013] An image processing system according to a second aspect of the present technology is an image processing system including the image processing device according to the first aspect of the present technology and a camera system having a camera.

[0014] FIG. 1 is a diagram for explaining operation of a wired PTZ camera. FIG. 2 is a diagram for explaining operation of a PTZ camera via a network. FIG. 3 is a diagram for explaining operation of a PTZ camera via a network. FIG. 4 is a diagram for explaining operation of a PTZ camera via a network. FIG. 5 is a diagram for explaining operation of a PTZ camera via a network. FIG. 6 is a diagram for explaining an example of the configuration of an image processing system. FIG. 7 is a diagram for explaining a specific example of cutout processing. FIG. 8 is a diagram for explaining a specific example of cutout processing. FIG. 9 is a flowchart for explaining shooting control processing. FIG. 10 is a flowchart for explaining shooting processing. FIG. 11 is a diagram for explaining an example of the configuration of a computer.

[0015] Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.

[0016] First Embodiment Example of Configuration of Image Processing System The present technology performs cutout processing on an image (video) captured by a camera based on an operation control signal indicating an operation amount for changing the camera's shooting range and a change amount signal indicating an amount of change in the actual shooting range, thereby making it possible to suppress the occurrence of discomfort due to delays when operating a camera.

[0017] First, the delay that occurs when a camera is remotely controlled will be described.

[0018] For example, consider an environment in which a PTZ camera CM11 and a controller CTL11 for controlling the PTZ camera CM11 are connected by wire, as shown in FIG.

[0019] In such an environment, for example, when an operator operates the controller CTL11 as shown by an arrow Q11, that operation is reflected on the PTZ camera CM11 side without delay as shown by an arrow Q12.

[0020] Therefore, the operator does not perceive any difference between the amount of operation he or she performs (operation amount) and the actual amount of movement of the PTZ camera CM11 or the camera head that moves in response to that operation. In other words, the operator can operate the PTZ camera CM11 without feeling any discomfort due to the delay until the operation he or she performs is actually reflected.

[0021] In contrast to this, consider an environment in which the PTZ camera CM11 and the controller CTL11 are connected via a network in which transmission delays may occur.

[0022] In such an environment, when an operator operates the controller CTL11, for example, as shown by arrow Q21 in Figure 2, an operation control signal indicating that operation is output from the controller CTL11. The operation control signal output from the controller CTL11 is then transmitted to the PTZ camera CM11 via the network NT11.

[0023] If a delay (transmission delay) occurs in the network NT11 when transmitting the operation control signal, the operation control signal has not yet been received by the PTZ camera CM11 immediately after the operator's operation, and so the operator's operation is not reflected, as shown by arrow Q22. In other words, the PTZ camera CM11 and the platform on which the PTZ camera CM11 is mounted have not moved.

[0024] After the operation control signal is output, when the PTZ camera CM11 receives the operation control signal, as shown by arrow Q31 in Figure 3, the PTZ camera CM11 and the camera platform on which the PTZ camera CM11 is installed actually move in accordance with the operation control signal, thereby reflecting the operator's operation.

[0025] Then, the image (video) captured after the movement, i.e., the image reflecting the operation, is transmitted (sent) from the PTZ camera CM11 to the operator via the network NT11. Even during this image transmission, delays may occur in the network NT11.

[0026] At this point, the image reflecting the operator's operation has not yet been received by the operator, so the image displayed on the operator's screen does not reflect the operator's operation, as shown by arrow Q32.

[0027] After that, when the image reflecting the operation is received by the operator, the image reflecting the operation will be displayed as shown by arrow Q41 in FIG.

[0028] In this way, if there is an uncontrollable delay between the controller CTL11 and the PTZ camera CM11, i.e., on the network NT11, when transmitting operation control signals or captured images, a delay will occur between the time the operator performs an operation and the time that operation is reflected in the image.

[0029] If such a delay becomes too great, it becomes difficult for the operator to grasp the extent to which his or her operation has been reflected, and as a result, it becomes difficult to perform the operation appropriately.

[0030] Therefore, in this technology, a device (unit) is provided on the operator side to assist with remote control (remote production).

[0031] This device reflects the operator's operations through cutout processing according to the amount of operation by the operator. That is, the device generates an image (video) that reflects the operator's operations and presents that image to the operator. This reduces the discomfort caused by delays when operating the camera.

[0032] FIG. 5 is a diagram showing an example of the configuration of an embodiment of an image processing system to which the present technology is applied.

[0033] The image processing system 11 shown in FIG. 5 includes a camera system 21, an image processing device 22, a controller 23, and a display / recording device 24.

[0034] For example, the camera system 21 is placed at the location where the content is shot, and the image processing device 22, controller 23, and display / recording device 24 are placed in a location different from the location, such as an office building, away from the location.

[0035] In the image processing system 11, the camera system 21 and the image processing device 22 are connected via a network 25, such as the Internet. The network 25 may be of any type, but for example, the network 25 may cause uncontrollable transmission delays.

[0036] The camera system 21 has at least a camera. For example, the camera system 21 may be configured with a PTZ camera that is capable of panning, tilting, and zooming, or may be configured with an arbitrary camera and a platform that allows the camera to be moved.

[0037] The camera system 21 includes a photographing unit 31 , a driving unit 32 , a sensor unit 33 , a control unit 34 , and an input / output unit 35 .

[0038] The photographing unit 31 includes, for example, a lens such as a zoom lens, an image sensor, and the like.

[0039] The photographing unit 31 photographs the surrounding subject in accordance with the control of the control unit 34, supplies the image signal of the resulting image (video signal of the video) to the control unit 34, and performs operations such as zooming in accordance with the control of the control unit 34.

[0040] For example, the shooting range of the shooting unit 31 changes due to the zoom operation of the shooting unit 31, i.e., the driving of the zoom lens. Note that hereinafter, an image obtained by shooting with the shooting unit 31 will also be referred to as a shot image.

[0041] The drive unit 32 consists of, for example, a motor provided on a camera or a camera head that constitutes the camera system 21, and by driving it in accordance with the control of the control unit 34, it changes the shooting range of the shooting unit 31, i.e., the range (area) in space that becomes the subject of the captured image.

[0042] For example, the driving unit 32 changes the shooting range of the shooting unit 31 (captured image) by translating the camera system 21, more specifically at least the shooting unit 31 of the camera system 21, based on the operation control signal. For example, the translation of the shooting unit 31 may be achieved by moving the shooting unit 31 itself, or by moving a platform to which a camera having the shooting unit 31 is fixed.

[0043] The direction of translation may be any of the horizontal direction, the vertical direction, and the direction parallel to the depth direction. The horizontal direction and the vertical direction are the left-right direction and the up-down direction as seen from the imaging unit 31, and the direction parallel to the depth direction is the direction perpendicular to the imaging surface of the image sensor of the imaging unit 31.

[0044] For example, the driving unit 32 rotates the camera system 21, more specifically at least the photographing unit 31 of the camera system 21, in a predetermined direction based on the operation control signal, thereby changing the photographing range of the photographing unit 31. The rotation of the photographing unit 31 may be achieved by rotating the photographing unit 31 itself, or by rotating a pan head to which a camera having the photographing unit 31 is fixed.

[0045] The rotation direction may be the rotation direction of any of pan, tilt, and roll operations. Rotation during panning occurs around a vertical line as an axis, rotation during tilting occurs around a horizontal line as an axis, and rotation during rolling occurs around a depth line as an axis.

[0046] Alternatively, a motor or the like that drives a zoom lens provided in the photographing unit 31 may function as the driving unit 32, and the driving unit 32 may realize the zoom operation.

[0047] The sensor unit 33 is composed of a sensor such as a linear sensor or a rotary encoder, detects the amount of change in the imaging range of the imaging unit 31, and supplies the detection result to the control unit 34. For example, the sensor unit 33 may be provided as part of the drive unit 32 or the imaging unit 31, or may be provided near the drive unit 32 or the imaging unit 31.

[0048] For example, the change in the shooting range detected by the sensor unit 33 may be the amount of parallel movement of the shooting unit 31, the rotation angle (amount of rotational movement) when the shooting unit 31 rotates, or the amount of change in zoom magnification (shooting magnification) due to the zoom operation of the shooting unit 31.

[0049] The control unit 34 controls the overall operation of the camera system 21 .

[0050] For example, the control unit 34 drives the drive unit 32 in response to an operation control signal for operating the camera system 21, supplied from the input / output unit 35, to change the shooting range of the shooting unit 31. The operation control signal is a control signal that indicates the amount of operation for changing the shooting range of the shooting unit 31 (camera). The operation here refers to, for example, a rotation operation, a translation operation, or a zoom operation.

[0051] Furthermore, for example, the control unit 34 generates a change amount signal indicating the amount of change in the shooting range of the shooting unit 31 based on the detection result supplied from the sensor unit 33, synchronizes the image signal of the captured image supplied from the shooting unit 31 with the change amount signal, and supplies the synchronized signal to the input / output unit 35. The change amount signal is a signal that indicates the actual amount of change in the shooting range (the amount by which the shooting range has changed) when the shooting range of the shooting unit 31 is changed, for example by moving the shooting unit 31 in response to the operation control signal.

[0052] The input / output unit 35 communicates with the image processing device 22 via the network 25. In particular, the input / output unit 35 functions as a transmission unit that transmits the image signal and change amount signal of the captured image supplied from the control unit 34 to the image processing device 22 via the network 25. In addition, for example, the input / output unit 35 receives an operation control signal transmitted from the image processing device 22 and supplies it to the control unit 34.

[0053] The image processing device 22 is a device (unit) for assisting in the remote production of content and the like.

[0054] For example, the image processing device 22 is connected to the controller 23 and the display recording device 24 via a wired or dedicated line, etc., so that no uncontrollable transmission delay occurs between the image processing device 22 and the controller 23 or the display recording device 24.

[0055] The image processing device 22 includes a control unit 51 and an image processing unit 52 .

[0056] The control unit 51 controls the operation of the camera system 21 by transmitting (outputting) various control signals for remotely controlling the camera system 21, such as operation control signals supplied from the controller 23, to the camera system 21 via the network 25. In particular, the control unit 51 functions as a transmission unit that transmits the operation control signals output from the controller 23 to the camera system 21.

[0057] Furthermore, when an operation control signal is supplied from the controller 23 , the control unit 51 supplies the operation control signal to the image processing unit 52 .

[0058] The image processing unit 52 receives the image signal of the captured image and the change amount signal transmitted from the camera system 21 via the network 25. In other words, the image processing unit 52 acquires the image signal of the captured image and the change amount signal synchronized with the image signal from the camera system 21 via the network 25.

[0059] In addition, the image processing unit 52 performs a cutout process to cut out a portion of the captured image based on the change amount signal and the operation control signal supplied from the control unit 51, and uses the image signal of the resulting image (video) as an image signal for the main line.

[0060] The image signal for the main line is an image signal of a target image, that is, an image to be obtained by shooting, such as an image signal of an image of content, etc. Hereinafter, the image for the main line obtained by the cutout process will also be referred to as a cut-out image.

[0061] The image processing unit 52 supplies (outputs) the image signal of the cut-out image (cut-out video) and the image signal of the captured image to the display / recording device 24 .

[0062] For simplicity of explanation, it is assumed here that the operation on the camera system 21 is at least one of translation, rotation, and zoom. However, the present invention is not limited to this, and an operation such as rotating the camera system 21 around a position different from the camera system 21 may also be performed. Even in such a case, an appropriate cut-out image can be obtained by cut-out processing based on the operation control signal and the change amount signal.

[0063] The controller 23 is operated by an operator who remotely operates the camera system 21, and supplies (outputs) various control signals, such as operation control signals according to the operator's operations, to the control unit 51 of the image processing device 22.

[0064] The display / recording device 24 is composed of an information processing device such as a computer placed near the operator, and receives image signals of captured images and cut-out images supplied from the image processing section 52 of the image processing device 22 .

[0065] The display / recording device 24 includes a display unit 81 and a recording unit 82 .

[0066] For example, the display unit 81 displays a captured image based on an image signal of the captured image, or displays a cut-out image based on an image signal of the cut-out image, while the recording unit 82 records the image signal of the captured image or the image signal of the cut-out image.

[0067] For example, in remote production, if the image output from the camera is output as is as an image for the main line, the operator will not be able to know the environment around the camera, i.e., what the subjects around the captured image are like, and it will not be easy to determine the appropriate direction to point the camera.

[0068] In contrast, in the image processing system 11, an image processing device 22 generates a cut-out image for the main line from the captured image, and not only the cut-out image but also a captured image with a wider angle of view than the cut-out image is output to the display recording device 24.

[0069] Therefore, the display recording device 24 displays the captured image on the display unit 81 and presents it to the operator, allowing the operator to grasp the subject outside the capture range of the cut-out image, which is the main line, i.e., the environment around the camera system 21. This allows the operator to operate the camera appropriately.

[0070] In this way, in the image processing system 11, the captured image (full-screen image) obtained by the camera system 21 is used as an image for checking the surrounding environment.

[0071] For simplicity of explanation, we will explain an example in which the captured image and the cut-out image are each output to the same display recording device 24, but the device to which the captured image is output and the device to which the cut-out image is output may be different devices.

[0072] 6 to 8, a specific example of the cutout processing performed by the image processing unit 52 of the image processing device 22 will be described. Note that in Fig. 6 to 8, corresponding parts are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

[0073] For example, suppose the operator performs a panning operation on the camera system 21 as indicated by arrow Q71 in FIG.

[0074] Then, the controller 23 outputs an operation control signal indicating the operation amount (operation amount) of the pan operation, i.e., the horizontal rotation angle (pan angle) for the pan operation, and this operation control signal is supplied to the camera system 21 via the control unit 51 and the network 25. The operation control signal is also supplied from the control unit 51 to the image processing unit 52.

[0075] For example, delays may occur in the network 25 when transmitting operation control signals. Therefore, in this example, the camera system 21 is not currently performing a panning operation, as indicated by arrow Q72. In other words, the operator's operation has not yet been reflected, and the horizontal rotation angle of the camera system 21 is 0 degrees.

[0076] In this example, it is assumed that a captured image P11, i.e., an image signal of the captured image P11, is supplied to the image processing unit 52. It is also assumed that the area to be cut out of the cut-out image immediately before the operation instructing a panning operation is cut-out area W11.

[0077] In this case, in the image processing unit 52, the cut-out area W11 is tilted in a direction parallel to the depth direction of the captured image by the pan angle indicated by the operation control signal, and the cut-out area W12 reflects the operator's operation, thereby obtaining a cut-out image P12 reflecting the operation.

[0078] That is, by performing a cutout process to cut out an image of the cutout region W12 obtained by adding perspective to the cutout region W11 by the pan angle, a cutout image P12 that reflects the operator's operation is obtained.

[0079] In this case, even if the operation has not yet been reflected in the camera system 21, which is the target of operation, the cut-out image P12 presented on the operator side reflects the operator's operation, so the operator can operate the camera system 21 without feeling any discomfort due to the delay.

[0080] 6, the operation has not yet been reflected in the camera system 21, so the amount of change indicated by the change amount signal output from the camera system 21, i.e., the amount of change in the horizontal rotation angle, is 0 degrees. Therefore, the cutout process is essentially being performed based only on the operation control signal.

[0081] When cutting out a cut-out image from a captured image, if the cut-out area is simply changed by the amount of operation indicated by the operation control signal, when the camera system 21 actually moves in response to the operation, the position of the cut-out area will deviate from the area intended by the operator.

[0082] Therefore, in the image processing system 11, a change amount signal synchronized with the image signal of the captured image is transmitted to the image processing device 22, and the position of the cutout region is adjusted based on the change amount signal.

[0083] For example, after the state shown in Figure 6, when the camera system 21 receives an operation control signal for a pan operation, the operation indicated by the operation control signal is reflected in the camera system 21, as shown by arrow Q81 in Figure 7. In other words, the camera system 21 rotates by the pan angle indicated by the operation control signal.

[0084] Then, the camera system 21 supplies (outputs) to the image processing unit 52 of the image processing device 22 an image signal of the captured image, along with a change amount signal indicating the angle at which the camera system 21 actually rotated, i.e., the amount of change in angle before and after rotation (amount of rotation).

[0085] 8, the image processing unit 52 performs cutout processing based on the change amount signal indicating the actual amount of rotation (amount of movement) of the camera system 21 and the operation control signal indicating the amount of operation by the operator. That is, the cutout processing is performed by adding perspective to the cutout area by an angle obtained by subtracting the actual amount of rotation from the amount of operation.

[0086] For example, suppose the angle of view of a captured image at a certain timing is area R31, and the cropped area at that time is area W31.

[0087] At this time, if the operator performs a panning operation so that the angle of view of the captured image becomes area R32, perspective is added to the cutout area according to the amount of operation (the rotation angle of the panning operation), and area W32 becomes the cutout area. In other words, cutout processing is performed to cut out area W32 and create the cutout image.

[0088] Furthermore, at the next timing, when the camera system 21 actually rotates in response to the operation control signal, the angle of view of the captured image changes from region R31 to region R33. As a result, the crop area also changes to region W33. However, since the angle of the crop area is actually adjusted by the amount of the change amount signal in the image processing unit 52, region W32 is ultimately used as the crop area. In other words, region W32 intended by the operator is cropped as the cropped image.

[0089] In this way, the cutout process is performed so as to obtain a desired angle of view by combining the amount of operation on the camera system 21 with the actual amount of movement of the camera system 21, i.e., the actual amount of change in the shooting range. By doing so, it is possible to prevent the operator from feeling a delay until the operation is reflected by utilizing the time difference between the operator's operation, i.e., the cutout process (cutout process), and the actual action. In other words, it is possible to suppress the occurrence of discomfort due to delays when operating the camera system 21 (when operating the camera).

[0090] <Explanation of Shooting Control Processing> Next, the operation of the image processing system 11 will be described.

[0091] When the operator operates the controller 23 to instruct the camera system 21 to start shooting, the controller 23 supplies (outputs) a control signal instructing the camera system 21 to start shooting to the control unit 51 of the image processing device 22. The image processing device 22 then performs the shooting control process shown in Fig. 9. The shooting control process performed by the image processing device 22 will be described below with reference to the flowchart of Fig. 9.

[0092] In step S11, the control unit 51 transmits the control signal supplied from the controller 23 to the camera system 21 via the network 25, thereby causing the camera system 21 to start capturing images.

[0093] When the camera system 21 starts taking pictures, the camera system 21 sequentially transmits (outputs) image signals of the captured images obtained by taking pictures and change amount signals synchronized with the image signals.

[0094] The operator also operates the controller 23 as appropriate while viewing the captured image displayed on the display unit 81. For example, the operator performs operations such as rotation, translation, and zoom as necessary on the camera system 21. Rotation operations include panning, tilting, and rolling, and translation operations include movement in the horizontal direction, vertical direction, and a direction parallel to the depth direction.

[0095] When the operator performs an operation such as a rotation operation or a parallel movement operation, an operation control signal indicating the operation is output from the controller 23 .

[0096] For example, when an operator performs an operation for a panning operation, i.e., a rotation operation in the horizontal direction, an operation control signal indicating the rotation angle in the horizontal direction instructed by the rotation operation is output. That is, an operation control signal having the rotation angle in the horizontal direction as the operation amount is output.

[0097] Furthermore, for example, when a zoom operation is performed by an operator, an operation control signal is outputted which indicates the zoom amount in the zoom operation, that is, the amount of change in the zoom magnification (amount of change in the focal length) as the operation amount.

[0098] In step S12 , the control unit 51 acquires an operation control signal from the controller 23 .

[0099] In step S13 , the control unit 51 transmits the operation control signal acquired from the controller 23 to the camera system 21 via the network 25 , and also supplies the acquired operation control signal to the image processing unit 52 .

[0100] If the operator does not perform any of the rotation operation, translation operation, and zoom operation on the camera system 21, an operation control signal indicating that the amount of operation is 0 is acquired in step S12. In other words, the process of acquiring and transmitting the operation control signal is not actually performed.

[0101] When an operation control signal is transmitted to the camera system 21, the camera system 21 performs an operation in accordance with the operation control signal, but depending on a transmission delay on the network 25, it may take some time for the operator's operation to be reflected in the camera system 21. In such a case, the amount of change indicated by the change amount signal may be zero.

[0102] In step S14, the image processing unit 52 receives the image signal of the captured image and the change amount signal synchronized with the image signal, both of which are transmitted from the camera system 21 via the network 25.

[0103] In step S15, the image processing unit 52 performs cutout processing based on the operation control signal supplied from the control unit 51 in step S13 and the change amount signal received in step S14, thereby generating a cut-out image.

[0104] In the cutout process, a partial area (cutout area) of the photographed image received in step S14 is cut out to form a cutout image.

[0105] In particular, in the cutout process, the image processing unit 52 changes the position and angle of the cutout area by the value obtained by subtracting the amount of change indicated by the change amount signal from the amount of operation indicated by the operation control signal.

[0106] In other words, the image processing unit 52 changes the position and angle of the cut-out area depending on the difference between the amount of change in the shooting range of the captured image indicated by the operation control signal and the amount of change in the shooting range of the captured image indicated by the change amount signal.

[0107] Specifically, when the operator performs a horizontal or vertical translation operation, the difference between the horizontal or vertical movement amount (change amount) of the camera system 21 (shooting range) indicated by the operation control signal and the actual horizontal or vertical movement amount of the camera system 21 (shooting range) indicated by the change amount signal is determined to be the difference in the change amount of the shooting range. In other words, the difference in the change amount of the shooting range is caused by the horizontal or vertical translation of the camera system 21. In this case, in the cutout process, the cutout area is moved horizontally or vertically by the difference in the change amount.

[0108] When the operator performs a translation operation in a direction parallel to the depth direction, the difference between the amount of movement (amount of change) of the camera system 21 (photography range) in the direction parallel to the depth direction indicated by the operation control signal and the amount of actual movement of the camera system 21 (photography range) in the direction parallel to the depth direction indicated by the change amount signal is determined to be the difference in the amount of change of the photography range. In this case, in the cutout process, the cutout area is enlarged or reduced by the amount of the difference in the amount of change.

[0109] When the operator performs a pan or tilt operation (rotation operation), the difference between the horizontal or vertical rotation angle (amount of change) of the camera system 21 (photography range) indicated by the operation control signal and the actual horizontal or vertical rotation angle (amount of rotation) of the camera system 21 (photography range) indicated by the change amount signal is regarded as the difference in the amount of change in the photography range. In other words, the difference in the amount of change in the photography range is caused by the pan or tilt operation of the camera system 21, i.e., the rotation in the horizontal or vertical direction.

[0110] In this case, in the cutout process, perspective is added to the cutout area by the difference in the amount of change. Specifically, the cutout area is tilted in a direction parallel to the depth direction of the captured image by an angle corresponding to the difference in the amount of change. In this way, when a pan or tilt operation is performed, cutout process taking perspective into account (perspective cutout process) is performed.

[0111] When the operator operates the roll (rotation operation), the difference between the roll rotation angle (amount of change) of the camera system 21 (capture range) indicated by the operation control signal and the actual roll rotation angle (amount of rotation) of the camera system 21 (capture range) indicated by the change amount signal is determined to be the difference in the amount of change in the capture range. The roll rotation angle here refers to the angle of rotation about a line perpendicular to the imaging surface of the imaging unit 31 of the camera system 21 as the rotation axis. In this case, in the cutout process, the cutout area is rotated about a line parallel to the depth direction of the captured image by an angle corresponding to the difference in the amount of change.

[0112] When the operator performs a zoom operation, the difference between the amount of change in the zoom magnification of the camera system 21 (photography range) indicated by the operation control signal and the amount of change in the actual zoom magnification of the camera system 21 (photography range) indicated by the change amount signal is determined as the difference in the amount of change in the photography range. In this case, in the cutout process, the cutout area is enlarged or reduced by the difference in the amount of change. In other words, the cutout image is resized.

[0113] In step S16, the image processing unit 52 outputs (transmits) to the display / recording device 24 the image signal of the captured image received in step S14 and the image signal of the cut-out image obtained in the cut-out process in step S15.

[0114] As a result, in the display / recording device 24 , the captured image or the cut-out image is displayed on the display unit 81 , and the image signal of the captured image or the image signal of the cut-out image is recorded in the recording unit 82 .

[0115] For example, the operator operates the controller 23 to appropriately change the angle of view of the captured image while checking the captured image displayed on the display unit 81. By displaying the captured image in this way, the operator can grasp the environment around the camera system 21 and more appropriately perform operations such as rotating and translating the camera system 21.

[0116] In step S16, the image processing unit 52 may output, instead of the image signal of the photographed image, an image signal of an image obtained by superimposing a frame indicating the cutout area during cutout processing on the photographed image, i.e., an image signal of the photographed image with the frame of the cutout area superimposed on it. In this way, the operator can grasp not only the environment around the camera system 21 but also which area has been designated as the cutout area.

[0117] In step S17, the control unit 51 determines whether or not to end the process of controlling image capture by the camera system 21. For example, in step S17, if a control signal instructing the end of image capture is supplied from the controller 23, it is determined that the process is to end.

[0118] If it is determined in step S17 that the process is not yet finished, the process then returns to step S12, and the above-described process is repeated.

[0119] On the other hand, if it is determined in step S17 that the process is to be ended, the control unit 51 stops the operation of each unit of the image processing device 22, and the photographing control process ends. In this case, the control unit 51 transmits the control signal supplied from the controller 23, which instructs the camera system 21 to end photographing, via the network 25, and ends photographing by the camera system 21.

[0120] In this manner, the image processing device 22 performs cutout processing based on the operation control signal and the change amount signal, and generates a cut-out image.

[0121] In this way, even if a delay occurs on the network 25, it is possible to suppress the occurrence of discomfort due to the delay when operating the camera system 21 (when operating the camera), thereby enabling the operator to perform operations comfortably without feeling stressed.

[0122] 9 is started in the image processing device 22, and when a control signal instructing the start of image capture is transmitted (output) from the image processing device 22 to the camera system 21, the camera system 21 performs the image capture process shown in Fig. 10. Hereinafter, the image capture process by the camera system 21 will be described with reference to the flowchart in Fig. 10.

[0123] In step S51, the control unit 34 controls the photographing unit 31 to start photographing.

[0124] 9, when a control signal is transmitted from the control unit 51, the input / output unit 35 receives the control signal transmitted by the control unit 51 and supplies it to the control unit 34. The control unit 34 then controls the photographing unit 31 to start photographing in accordance with the control signal supplied from the input / output unit 35. The photographing unit 31 photographs images under the control of the control unit 34, and sequentially supplies image signals of the photographed images obtained by photographing to the control unit 34.

[0125] In step S52 , the control unit 34 acquires the operation control signal output from the controller 23 .

[0126] That is, when the operator performs an operation such as a rotation operation, a translation operation, or a zoom operation on the camera system 21, the control unit 51 transmits (outputs) an operation control signal corresponding to the operator's operation in step S13 of Fig. 9. The input / output unit 35 receives the operation control signal transmitted from the control unit 51 via the network 25, and supplies the received operation control signal to the control unit 34. In this way, the control unit 34 acquires the operation control signal.

[0127] In step S53, the control unit 34 controls the driving unit 32 or the photographing unit 31 to perform driving in accordance with the operation control signal acquired in step S52.

[0128] For example, when the operation control signal instructs translation of the camera system 21, i.e., the image capturing unit 31, the control unit 34 supplies the operation control signal to the drive unit 32. In this case, the drive unit 32 is driven based on the operation control signal supplied from the control unit 34, and translates the camera platform to which the camera having the image capturing unit 31 is fixed by the amount of translation indicated by the operation control signal, thereby translating the image capturing range of the image capturing unit 31.

[0129] Furthermore, for example, when an operation control signal instructs rotation of the camera system 21 (image capture unit 31), the control unit 34 supplies the operation control signal to the drive unit 32. In this case, the drive unit 32 is driven based on the operation control signal supplied from the control unit 34, and rotates the camera platform to which the camera having the image capture unit 31 is fixed, or the image capture unit 31 itself, by an amount of operation indicated by the operation control signal, thereby rotating the image capture range of the image capture unit 31.

[0130] Furthermore, for example, when an operation control signal instructs the camera system 21 (photographing unit 31) to zoom, the control unit 34 changes the photographing range of the photographing unit 31 by, for example, moving the zoom lens that constitutes the photographing unit 31 based on the operation control signal.

[0131] When the drive unit 32 or the photographing unit 31 is driven and the photographing range of the photographing unit 31 changes, a sensor signal indicating the detection result of the change in the photographing range, i.e., the detection result of the amount of movement or rotation of the photographing unit 31, the amount of change in the zoom magnification, etc., is output from the sensor unit 33.

[0132] In step S54, the control unit 34 generates a change amount signal based on the sensor signal output from the sensor unit 33, which indicates the amount of change in the shooting range by the shooting unit 31, i.e., the actual amount of change in the position, angle (orientation), zoom magnification, etc. of the shooting unit 31.

[0133] In step S55, the control unit 34 synchronizes the image signal of the captured image supplied from the imaging unit 31 with the change amount signal generated in step S54 and supplies them to the input / output unit 35, causing the input / output unit 35 to transmit the image signal of the captured image and the change amount signal. The input / output unit 35 transmits (outputs) the image signal of the captured image and the change amount signal supplied from the control unit 34 to the image processing device 22 via the network 25.

[0134] In step S56, the control unit 34 determines whether or not to end the process of capturing an image. For example, the control unit 34 determines to end the process when a control signal instructing the end of image capturing, transmitted from the image processing device 22, is received by the input / output unit 35 and supplied to the control unit 34.

[0135] If it is determined in step S56 that the process is not yet finished, the process then returns to step S52, and the above-described process is repeated.

[0136] On the other hand, if it is determined in step S56 that the process is to be ended, the control unit 34 stops the operation of each unit of the camera system 21, and the photographing process ends.

[0137] As described above, the camera system 21 operates in response to the operation control signal, and transmits (outputs) a change amount signal together with an image signal of a captured image.

[0138] In this way, the image processing device 22, i.e., the operator, performs cutout processing based on the operation control signal and the change amount signal, and a cutout image that reflects the operator's operation can be obtained, thereby suppressing the occurrence of discomfort due to delays.

[0139] According to the present technology as described above, even in an environment where uncontrollable delays may occur between the controller 23 and the camera system 21, a cropped image that instantly reflects the operation of the controller 23 by the operator can be obtained. Furthermore, after the camera system 21 operates (moves) appropriately in response to the operator's operation, an appropriate area is cropped as the cropped image based on the relationship between the amount of operation instructed by the operator and the amount of change when the camera system 21 actually moves. As a result, the operator can perform operations without feeling the effects of delays, i.e., without feeling any discomfort due to the delays.

[0140] The above describes an example in which a cut-out image is generated in real time when the image is captured, reflecting the operator's operations in the image processing device 22. However, the present invention is not limited to this. The captured image, more specifically, the image signal of the captured image may be recorded once, and the operator may perform operations such as panning, tilting, and translation on the recorded captured image, and cut-out processing may be performed on the captured image based on an operation control signal indicating the results of the operation.

[0141] <Example of Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that constitute the software are installed on a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose computer, for example, that can execute various functions by installing various programs.

[0142] FIG. 11 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0143] In the computer, a CPU (Central Processing Unit) 501 , a ROM (Read Only Memory) 502 , and a RAM (Random Access Memory) 503 are interconnected by a bus 504 .

[0144] An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

[0145] The input unit 506 includes a keyboard, a mouse, a microphone, an image sensor, etc. The output unit 507 includes a display, a speaker, etc. The recording unit 508 includes a hard disk, a non-volatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0146] In a computer configured as described above, the CPU 501 loads a program recorded in the recording unit 508, for example, into the RAM 503 via the input / output interface 505 and the bus 504, and executes the program, thereby performing the above-described series of processes.

[0147] The program executed by the computer (CPU 501) can be provided by being recorded on a removable recording medium 511 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0148] In a computer, a program can be installed in the recording unit 508 via the input / output interface 505 by inserting a removable recording medium 511 into the drive 510. The program can also be received by the communication unit 509 via a wired or wireless transmission medium and installed in the recording unit 508. Alternatively, the program can be installed in the ROM 502 or the recording unit 508 in advance.

[0149] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0150] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0151] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0152] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0153] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0154] Furthermore, the present technology can also be configured as follows.

[0155] (1) An image processing device including an image processing unit that performs cutout processing to cut out a part of an image captured by a camera based on an operation control signal indicating an operation amount for changing a capturing range of the camera and a change amount signal indicating an amount of change in the capturing range. (2) The image processing device described in (1), in which the image processing unit changes the area to be cut out by the cutout processing in accordance with the difference between the amount of change in the capturing range indicated by the operation control signal and the amount of change in the capturing range indicated by the change amount signal. (3) The image processing device described in (2), in which the image processing unit changes the area in accordance with the difference in the amount of change in the capturing range due to translation of the camera. (4) The image processing device described in (3), in which the image processing unit moves the area horizontally or vertically in accordance with the difference in the amount of change in the capturing range in the horizontal or vertical direction. (5) The image processing device described in (3) or (4), in which the image processing unit enlarges or reduces the area in accordance with the difference in the amount of change in the capturing range in a direction parallel to the depth direction. (6) The image processing device according to any one of (2) to (5), wherein the image processing unit changes the region according to the difference in the amount of change in the shooting range due to rotation of the camera. (7) The image processing device according to (6), wherein the image processing unit performs the cutout processing by adding perspective to the region by an angle according to the difference in the amount of change in the shooting range due to panning or tilting of the camera. (8) The image processing device according to (6) or (7), wherein the image processing unit rotates the region by an angle according to the difference in the amount of change in the shooting range due to roll of the camera. (9) The image processing device according to any one of (2) to (8), wherein the image processing unit enlarges or reduces the region according to the difference in the amount of change in the shooting range due to zooming of the camera. (10) The image processing device according to any one of (2) to (9), wherein the image processing unit outputs an image signal of the image and an image signal of a cut-out image obtained by the cutout processing. (11) The image processing device according to (10), wherein the image processing unit outputs an image signal of an image obtained by superimposing a frame indicating the area on the image captured by the camera.(12) The image processing device according to any one of (1) to (11), wherein the image processing device is connected to a camera system having at least the camera via a network. (13) The image processing device according to (12), further comprising a transmitter that transmits the operation control signal to the camera system via the network. (14) The image processing device according to (12) or (13), wherein the image processing unit acquires an image signal of the image and the change amount signal synchronized with the image signal from the camera system via the network. (15) An image processing method, wherein the image processing device performs cutout processing to cut out a part of an image captured by the camera based on an operation control signal that indicates an operation amount for changing a shooting range of a camera and a change amount signal that indicates an amount of change in the shooting range. (16) An image processing system having a camera system having at least a camera and an image processing device connected via a network, wherein the image processing device comprises: a first transmission unit that transmits an operation control signal to the camera system indicating an operation amount for changing the shooting range of the camera; and an image processing unit that performs a cutout process to cut out a part of an image shot by the camera based on the operation control signal and a change amount signal indicating an amount of change in the shooting range; and the camera system comprises: a drive unit that changes the shooting range of the camera based on the operation control signal; and a second transmission unit that transmits an image signal of the image and the change amount signal to the image processing device.

[0156] REFERENCE SIGNS LIST 11 Image processing system, 21 Camera system, 22 Image processing device, 23 Controller, 25 Network, 31 Photography unit, 32 Drive unit, 33 Sensor unit, 34 Control unit, 35 Input / output unit, 51 Control unit, 52 Image processing unit

Claims

1. An image processing device comprising an image processing unit that performs cutout processing to cut out a part of an image captured by a camera based on an operation control signal that indicates the amount of operation for changing the camera's shooting range and a change amount signal that indicates the amount of change in the shooting range.

2. The image processing device according to claim 1, wherein the image processing unit changes the area cut out by the cutout processing in accordance with the difference between the amount of change in the shooting range indicated by the operation control signal and the amount of change in the shooting range indicated by the change amount signal.

3. The image processing device according to claim 2, wherein the image processing unit changes the area in accordance with the difference in the amount of change in the imaging range due to translation of the camera.

4. The image processing device according to claim 3, wherein the image processing unit moves the area horizontally or vertically in accordance with the difference in the amount of change in the shooting range in the horizontal or vertical direction.

5. The image processing device according to claim 3, wherein the image processing unit enlarges or reduces the area according to the difference in the amount of change in the direction parallel to the depth direction of the imaging range.

6. The image processing device according to claim 2, wherein the image processing unit changes the area in accordance with the difference in the amount of change in the imaging range due to rotation of the camera.

7. The image processing device according to claim 6, wherein the image processing unit performs the cutout processing by adding perspective to the area by an angle corresponding to the difference in the amount of change in the shooting range due to panning or tilting of the camera.

8. The image processing device according to claim 6, wherein the image processing unit rotates the area by an angle corresponding to the difference in the amount of change in the shooting range due to the roll of the camera.

9. The image processing device according to claim 2, wherein the image processing unit enlarges or reduces the area according to the difference in the amount of change in the shooting range due to the zoom of the camera.

10. The image processing device according to claim 2, wherein the image processing section outputs an image signal of the image and an image signal of the cut-out image obtained by the cut-out processing.

11. The image processing device according to claim 10, wherein the image processing section outputs an image signal of an image obtained by superimposing a frame indicating the area on the image captured by the camera.

12. The image processing device according to claim 1, wherein the image processing device is connected to a camera system having at least the camera via a network.

13. The image processing device according to claim 12, further comprising a transmission unit that transmits the operation control signal to the camera system via the network.

14. The image processing device according to claim 12, wherein the image processing unit acquires the image signal of the image and the change amount signal synchronized with the image signal from the camera system via the network.

15. An image processing method in which an image processing device performs cutout processing to cut out a part of an image captured by a camera based on an operation control signal indicating the amount of operation for changing the shooting range of the camera and a change amount signal indicating the amount of change in the shooting range.

16. An image processing system comprising a camera system having at least a camera and an image processing device connected via a network, wherein the image processing device comprises: a first transmission unit that transmits to the camera system an operation control signal indicating an operation amount for changing the shooting range of the camera; and an image processing unit that performs cutout processing to cut out a part of an image shot by the camera based on the operation control signal and a change amount signal indicating the amount of change in the shooting range; and the camera system comprises: a drive unit that changes the shooting range of the camera based on the operation control signal; and a second transmission unit that transmits an image signal of the image and the change amount signal to the image processing device.

Images