Imaging apparatus and method for controlling imaging apparatus

Abstract

To provide a technique to effectively use a scaling window so as to prevent an increase in the amount of codes of a photographed image, in an imaging apparatus that allows control of a photographing direction.SOLUTION: To control the position of a scaling window for signalling the center of a reference image used for inter-prediction in encoding of a photographed image, according to the amount of movement of an imaging apparatus in a photographing direction during control of the photographing direction.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to a technique for encoding captured images. [Background technology]

[0002] In recent years, with the expansion of the video distribution market, there has been an increase in filming systems for filming events such as weddings and lectures using cameras that can be remotely controlled and shoot video over a network. While remotely controlled cameras are generally operated using a hardware controller, methods for controlling PTZ using an application (software controller) that runs on a PC or mobile device have also been realized. In addition, applications can receive and play back video images captured and distributed by the camera.

[0003] Cameras encode video data to efficiently distribute it over a network and store it in storage. While H.264 and H.265 are well-known encoding methods, new encoding methods have been proposed to meet various new requirements. It is known that encoding efficiency drops particularly during panning, tilting, and zooming, but encoding methods that do not drop in encoding efficiency even in these situations have also been proposed.

[0004] The Versatile Video Coding (VVC) coding method (hereafter referred to as VVC) is known as a coding method for compressing and recording moving images. To improve coding efficiency, VVC incorporates a technology called Reference Picture Resampling (RPR). RPR allows an image with a different resolution from the image being decoded to be used as a reference image, enabling resolution changes even in inter-frame compression. Furthermore, the RPR resolution ratio can be expressed as a signal called a scaling window, and by properly controlling this, the amount of code can be effectively reduced during pan-tilt and zoom operations.

[0005] In the technology disclosed in Patent Document 1, a PTZ camera uses four types of information, namely, the size of the sensor pixels, the focal length, PT information, and multiple vertices for the mask, to calculate angle information for the multiple vertices that form the area to be masked after PT is performed.Then, the multiple vertices that form the area to be masked on the sensor are calculated, and the mask area is set. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent 03996805 Summary of the Invention [Problem to be solved by the invention]

[0007] However, in the conventional technology disclosed in the above-mentioned Patent Document 1, only a mask area is calculated and set using PT information, etc., and it is not possible to calculate and set a scaling window control value, which will be described later. Therefore, it is not possible to expect an effect of improving encoding efficiency. The present invention provides a technology for effectively using a scaling window so as not to increase the code amount of a captured image in an imaging device capable of controlling the shooting direction. [Means for solving the problem]

[0008] One aspect of the present invention is an imaging device comprising: an encoding means for encoding a captured image; and a control means for controlling the position of a scaling window that signals the center of a reference image to be used for inter-prediction in the encoding, in accordance with the amount of movement of the imaging direction when the imaging direction of the imaging device is controlled. [Effects of the Invention]

[0009] According to the present invention, in an imaging device capable of controlling the imaging direction, a scaling window can be effectively used so that the code amount of a captured image does not become large. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 9 is a block diagram showing an example of the configuration of an encoding unit 905. [Figure 2] FIG. 10 is a diagram illustrating how to set a scaling window. [Figure 3] FIG. 10 is a diagram showing an example of a table configuration. [Figure 4] FIG. 10 is a diagram showing an example of a table configuration. [Figure 5] FIG. 10 is a diagram showing an example of a table configuration. [Figure 6] A simplified diagram to easily explain the thinning method. [Figure 7] A simplified diagram to easily explain the thinning method. [Figure 8] FIG. 10 is a diagram illustrating a motion vector when a scaling window is set. [Figure 9] FIG. 11 is a block diagram showing an example of the hardware configuration of an imaging device 1101. [Figure 10] 10 is a flowchart of a process related to setting a scaling window. [Figure 11] FIG. 1 is a diagram showing an example of a system configuration. [Figure 12] FIG. 10 is a diagram showing a specific example of a process for calculating a scaling window control value. [Figure 13] FIG. 10 is a diagram showing a specific example of a process for calculating a scaling window control value. [Figure 14] FIG. 10 is a diagram showing a specific example of a process for calculating a scaling window control value. [Figure 15] FIG. 10 is a diagram showing a specific example of a process for calculating a scaling window control value. [Figure 16] FIG. 10 is a diagram showing a specific example of a process for calculating a scaling window control value. [Figure 17] 11 is a flowchart of the overall operation of the imaging device 1101. DETAILED DESCRIPTION OF THE INVENTION

[0011] The following describes embodiments in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Although the embodiments describe multiple features, not all of these features are necessarily essential to the invention, and multiple features may be combined in any desired manner. Furthermore, in the accompanying drawings, the same or similar components are designated by the same reference numerals, and redundant description will be omitted. Note that in the following description, "signaling" refers to encoding target information, parameters, flags, etc. into coded data or a bitstream. In this way, the values ​​of the information, parameters, flags, etc. can be transmitted to a decoding device.

[0012] [First embodiment] First, an example of the configuration of a system according to this embodiment will be described with reference to Fig. 11. As shown in Fig. 11, the system according to this embodiment includes an image capture device 1101 such as a network camera or an IP streaming camera, and terminal devices 1102a and 1102b such as PCs and tablet terminals. The image capture device 1101 and the terminal devices 1102a and 1102b are connected to a network 1103, thereby enabling data communication between them. Hereinafter, the terminal devices 1102a and 1102b will be collectively referred to as the terminal devices 1102.

[0013] In response to a command received from the terminal device 1102, the imaging device 1101 transmits captured images and various information to the terminal device 1102, and changes the pan, tilt, zoom, focus, etc. of the imaging device 1101. Note that the imaging device 1101 may transmit captured images and various information to the terminal device 1102 regardless of whether a command has been received.

[0014] The terminal device 1102 generates various commands in response to user operations and transmits the generated commands to the imaging device 1101. The terminal device 1102 can then display or store captured images and various information received from the imaging device 1101 in response to the transmission.

[0015] The network 1103 is realized by, for example, a plurality of routers, switches, cables, etc. that comply with a communication standard such as ETHERNET (registered trademark). Note that the network 1103 may also be realized by the Internet, a wired LAN (Local Area Network), a wireless LAN, a WAN (Wide Area Network), etc.

[0016] Next, an example of the hardware configuration of the imaging device 1101 will be described using the block diagram in Fig. 9. The control unit 901 executes various processes using computer programs and data stored in the memory unit 907. In this way, the control unit 901 controls the overall operation of the imaging device 1101, and also executes or controls various processes that will be described as processes performed by the imaging device 1101.

[0017] For example, the control unit 901 analyzes a camera control command transmitted from the terminal device 1102 via the network 1103 and received by the communication unit 908, and performs processing according to the results of the analysis. Here, the camera control commands can be roughly classified into request commands for requesting acquisition of captured images or setting values ​​of the imaging device 1101, and setting commands for requesting the imaging device 1101 to set setting values.

[0018] When the control unit 901 receives a "request command for requesting acquisition of a captured image" from the terminal device 1102, the control unit 901 transmits the bit stream generated by the encoding unit 905 to the terminal device 1102 via the communication unit 908 and the network 1103. The control unit 901 can also store the bit stream in the storage unit 906.

[0019] Also, suppose that the control unit 901 receives a "request command requesting acquisition of the zoom, focus, pan, and tilt setting values ​​of the imaging device 1101" from the terminal device 1102. At this time, the control unit 901 acquires the "zoom and focus setting values" and the "pan and tilt (image capture direction) setting values" from the imaging unit 902 and the pan / tilt control unit 903, respectively, and transmits the acquired setting values ​​to the terminal device 1102 via the network 1103 using the communication unit 908. Here, the setting values ​​of zoom, focus, pan, and tilt include not only the current values ​​but also related information such as the range of values ​​that can be set in the imaging device 1101.

[0020] Also, it is assumed that the control unit 901 receives a "setting command to set the zoom, focus, pan, and tilt setting values" from the terminal device 1102. At this time, the imaging unit 902, under the control of the control unit 901, changes the current zoom and focus in accordance with the "zoom and focus setting values" in the setting command. Similarly, under the control of the control unit 901, the pan / tilt control unit 903 changes the current pan and tilt in accordance with the "pan and tilt setting values" in the setting command. As a result, the setting values ​​for zoom, focus, pan, and tilt set by the terminal 1102 are reflected in the imaging device 1101. Note that a detailed description of zoom and focus will be omitted.

[0021] The imaging unit 902 includes an optical system including multiple lenses, an imaging element, respective drive control circuits, and various image processing circuits. The imaging element outputs an analog image signal corresponding to an optical image received via the optical system, and the image processing circuit performs various processes on the analog image signal, such as A / D conversion, development processing, various color processing, and resolution conversion processing, to generate a captured image. The imaging unit 902 may be an imaging unit that captures moving images, or may be an imaging unit that periodically or irregularly captures still images. If the imaging unit 902 is an imaging unit that captures moving images, the imaging unit 902 generates images of each frame in the captured moving images as captured images. On the other hand, if the imaging unit 902 is an imaging unit that periodically or irregularly captures still images, the imaging unit 902 generates the still images as captured images.

[0022] The pan / tilt control unit 903 has drive systems for pan and tilt and motors for driving the drive systems. Under the control of the control unit 901, the pan / tilt control unit 903 controls the pan and tilt positions of the imaging unit 902 based on "pan and tilt setting values." Here, the "pan and tilt setting values" are expressed as angles (degrees) shifted from a predetermined pan / tilt origin.

[0023] The scaling window control unit 904 controls the scaling window control value. Details of the scaling window control unit 904 will be described later. The encoding unit 905 encodes the captured image generated by the imaging unit 902, generates a bit stream including the encoding result of the captured image, and transmits the generated bit stream to the terminal device 1102 via the communication unit 908 and the network 1103. Details of the encoding unit 905 will be described later.

[0024] The storage unit 906 is a non-volatile memory device that functions as internal storage of the imaging device 1101 and / or external storage of the imaging device 1101. For example, a memory device such as an HDD, SSD, or FROM can be used as the internal storage of the imaging device 1101 or the external storage of the imaging device 1101. The storage unit 906 stores computer programs and data for causing the control unit 901 to execute or control the overall operation of the imaging device 1101. The computer programs and data stored in the storage unit 906 are loaded into the memory unit 907 as appropriate under the control of the control unit 901, and become targets for processing by the control unit 901.

[0025] The memory unit 907 is a volatile memory device such as a RAM. The memory unit 907 has an area for storing computer programs and data loaded from the storage unit 906, and an area for storing information such as commands received from the outside via the communication unit 908. The memory unit 907 also has an area for storing captured images generated by the imaging unit 902. The memory unit 907 also has work areas used by the control unit 901, scaling window control unit 904, and encoding unit 905 when performing various processes. In this way, the memory unit 907 can provide various areas as needed.

[0026] The communication unit 908 performs data communication with the terminal device 1102 via the network 1103. For example, the communication unit 908 transmits a bit stream generated by the imaging device 1101 to the terminal device 1102 via the network 1103. Also, for example, the communication unit 908 receives a command transmitted from the terminal device 1102 via the network 1103, and transmits a response generated by the imaging device 1101 in response to the command to the terminal device 1102 via the network 1103.

[0027] Next, an example configuration of the encoding unit 905 will be described using the block diagram of Fig. 1. The functional units shown in Fig. 1 may be implemented by hardware or software (computer program). In the latter case, the software is executed by the control unit 901, thereby realizing the functions of the corresponding functional units. The configuration shown in Fig. 1 may also be implemented by a combination of hardware and software. The encoding unit 905 acquires a captured image as an input image, and encodes the input image using VVC to generate a bitstream.

[0028] The image analysis unit 102 acquires a captured image to be encoded as an input image, analyzes the angle of view change value and optical axis change value of the input image, and generates the analysis results as image analysis information. The image analysis unit 102 also divides the input image into multiple tile images based on the image characteristics of the input image and external inputs.

[0029] The generation unit 103 generates scaling window control information (RPR control information) including information on the scaling ratio and offset position of the motion vector required for decoding using RPR. The scaling window is a technology standardized by VVC, and is used for efficient coding by changing the scaling window in synchronization with changes in the angle of view when the zoom is changed, etc.

[0030] The prediction unit 104 performs prediction processing such as intra-prediction, which is intra-frame prediction, or inter-prediction, which is inter-frame prediction, on the tile images to generate predicted images, and calculates prediction errors from the tile images and the predicted images. The prediction unit 104 also generates information necessary for the prediction processing, such as prediction modes and motion vectors, as prediction information.

[0031] The transform / quantization unit 105 performs orthogonal transform on the prediction errors in units of blocks to generate transform coefficients, and quantizes the transform coefficients to generate quantized coefficients. The inverse quantization / inverse transform unit 106 inverse quantizes the quantized coefficients generated by the transform / quantization unit 105 to regenerate transform coefficients, and further performs inverse orthogonal transform on the regenerated transform coefficients to regenerate the prediction errors.

[0032] The frame memory 108 is a memory for storing a reproduced image (reproduced image). The image reproduction unit 107 generates a predicted image by appropriately referring to the frame memory 108 based on the prediction information generated by the prediction unit 104, generates a reproduced image from the generated predicted image and a prediction error, and stores the reproduced image in the frame memory 108.

[0033] The in-loop filter unit 109 generates a filtered reconstructed image by performing in-loop filtering such as deblocking filtering and sample adaptive offset on the reconstructed image stored in the frame memory 108. The in-loop filter unit 109 then stores the generated filtered reconstructed image in the frame memory 108.

[0034] The entropy coding unit 110 encodes the quantization coefficients generated by the transform / quantization unit 105 and the prediction information generated by the prediction unit 104 to generate coded data.

[0035] The integrated coding unit 111 generates header code data by encoding the image analysis information generated by the image analysis unit 102 and the scaling window control information generated by the generation unit 103. The integrated coding unit 111 then generates a bit stream including the header code data and the code data generated by the entropy coding unit 110, and outputs the generated bit stream.

[0036] Next, the operation performed by the encoding unit 905 having such a configuration to encode one frame of input image (sometimes simply referred to as a frame) will be described in more detail.

[0037] The image analysis unit 102 calculates the angle of view change value and optical axis change value for the frame. When an arbitrary frame is set as a reference frame, the angle of view change value may be the ratio of the angle of view of the reference frame to the frame to be encoded. Furthermore, when the center of the reference frame is set as the optical axis, the optical axis change value is a numerical value that represents the deviation of the image between the frame to be encoded and the reference frame. For example, PTZ information may be input as an external input, and the angle of view change value and optical axis change value may be calculated before calculating the scaling window control value.

[0038] The generation unit 103 signals the use of RPR by setting sps_ref_pic_resampling_enabled_flag in the Sequence Parameter Set (hereinafter, SPS) to 1. The generation unit 103 also stores the number of horizontal and vertical luminance pixels of the frame as pps_pic_width_in_luma_samples and pps_pic_height_in_luma_samples in the Picture Parameter Set (hereinafter, PPS), respectively.

[0039] The generation unit 103 also stores scaling window control values ​​in the PPS. Here, the scaling window is a rectangular area defined by the following scaling window control values: pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, and pps_scaling_win_bottom_offset. The scaling window control values ​​are expressed in terms of the number of samples of an image consisting of color-difference signals, and represent the offset values ​​of the four sides of the scaling window from the four sides of the frame containing the scaling window. If these values ​​are not explicitly specified, they are treated as 0, and the frame containing the scaling window itself becomes the scaling window. Each control value can be set within the following range. Unless otherwise specified, the image format is set to YUV444, and the coordinate values ​​indicating the scaling window control value and the coordinate values ​​indicated by the number of luminance samples are set to the same value. For example, if the image format is YUV420, the coordinate values ​​indicated by the scaling window control value must be doubled to match the coordinate values ​​indicated by the number of luminance samples. In addition, in the VVC standard, the scaling window control value must satisfy the following relationship:

[0040] pps_scaling_win_X_offset <Width … (1) pps_scaling_win_X_offset≧-15xWidth … (2) Here, pps_scaling_win_X_offset means either pps_scaling_win_left_offset (hereinafter referred to as SWl_offset) or pps_scaling_win_right_offset (hereinafter referred to as SWr_offset). Width is the number of horizontal samples of the luminance value of the image expressed by pps_pic_width_in_luma_samples.

[0041] Similarly, with the number of samples in the vertical direction pps_pic_height_in_luma_samples as Height, pps_scaling_win_Y_offset is set within the ranges shown in the following equations (3) and (4).

[0042] pps_scaling_win_Y_offset <Height … (3) pps_scaling_win_Y_offset≧-15xHeight … (4) Here, pps_scaling_win_Y_offset means either pps_scaling_win_top_offset (hereinafter referred to as SWt_offset) or pps_scaling_win_bottom_offset (hereinafter referred to as SWb_offset). In addition, the following relationship must also be satisfied.

[0043] -15xWidth≦SWl_offset+SWr_offset <Width … (5) -15xHeight≦SWt_offset+SWb_offset <Height … (6) The VVC standard allows the scaling window to be freely set within a range that satisfies the above formulas (1) to (6). Here, a method for setting the scaling window according to this embodiment will be described with reference to FIG.

[0044] In this embodiment, the angle of view and resolution change simultaneously during zooming. Therefore, the reference frame used to set the scaling window is set to the frame after zooming, and setting of the scaling window is started in a frame (e.g., an IDR frame) that does not use RPR before zooming begins. In Figure 2, reference numbers 201 and 202 indicate frames before zooming begins, reference numbers 203 and 204 indicate frames during zooming, and reference number 205 indicates a frame after zooming ends.

[0045] When the imaging device 1101 receives a zoom operation, it generates a frame 201 that does not use RPR and sets a scaling window 206 for the frame 201. The scaling window 206 is the range that is captured in a frame 205 after the zoom operation is completed, calculated from the angle of view after the zoom operation.

[0046] Furthermore, the imaging device 1101 generates a high-resolution frame 202 that references frame 201 using RPR as a frame before ZOOM begins, and sets a scaling window 207 for this frame 202 in the same manner as scaling window 206. Here, the resolution of frame 202 may be determined so that the number of pixels in scaling window 207 is equal to the resolution of frame 205 after ZOOM ends. After ZOOM begins, frames may be generated at the same resolution as frame 201. For example, the size of scaling window 208 for frame 203 is larger than scaling window 206 because ZOOM has started. Frame 204 is also generated at the same resolution as frame 201, and similarly, scaling window 209 for frame 204 is larger than scaling window 208. After ZOOM ends, frame 205 is generated at the same resolution as the resolution within scaling window 207, and since the scaling window covers the entire image, no setting is required. In this way, frames that do not use RPR, which requires a large amount of coding to generate a scaling window, can be generated at a lower resolution.

[0047] 1, the prediction unit 104 divides the tile images divided by the image analysis unit 102 into multiple blocks, and performs prediction processing on a block-by-block basis to generate prediction errors and prediction information. Here, the prediction processing performed by the prediction unit 104 and the prediction information generated by the prediction unit 104 will be described in more detail.

[0048] Image coding technologies such as VVC use a prediction process to predict the pixels of a block to be coded using pixels from a previously coded block in order to reduce the data volume of the coded bitstream while maintaining the image quality of the reproduced image. Prediction processes include intra-prediction, which uses pixels from a previously coded block in the same frame, and inter-prediction, which uses pixels from a block in a different coded frame. In addition, VVC standardizes a technique called RPR to enable decoding even when the resolution of the previously coded frame to be referenced and the frame to be coded are different.

[0049] Here, as an explanation of RPR, inter prediction when the resolution of a reference encoded frame and the encoding target frame differs will be further explained. The reference encoded frame is scaled to match the resolution of the encoding target frame, and inter prediction is performed. An example of a method for scaling the reference encoded frame is shown below. For simplicity, the explanation will be given for luminance values. As for chrominance, a similar discussion can be applied taking into account the number of samples, and therefore an explanation will be omitted. The prediction unit 104 is responsible for the following steps A1, A2, and A3.

[0050] (Step A1) The vertical scaling ratio scalingRatio[0] and the horizontal scaling ratio scalingRatio[1] are calculated. Hereinafter, scalingRatio[0] and scalingRatio[1] are collectively referred to as scalingRatio[x]. scalingRatio[x] is determined by the ratio between the size of the scaling window of the coded frame to be referenced and the size of the scaling window of the current frame to be coded. In this embodiment, the prediction unit 104 obtains this scalingRatio[x] from the generation unit 103 as RPR control information.

[0051] (Step A2) This function determines the interpolation filter used for scaling. For example, the coefficients of the interpolation filter are selected depending on the value of scalingRatio[x]. If scalingRatio[x] exceeds 1.75x, the coefficients in the table in Figure 3 are used. If scalingRatio[x] is less than 1.75x but exceeds 1.25x, the coefficients in the table in Figure 4 are used. In all other cases, the coefficients in the table in Figure 5 are used. The coefficients of the interpolation filter are determined by the sample position p to be calculated. p is an integer between 0 and 15, and is the numerator value when the minimum sample unit is divided into 1 / 16ths. For example, if sample point A and sample point (A+1) are divided into 16, the filter coefficients for the third sample point (A+3 / 16, p=3) are fL[3][i]=[-4, -1, 16, 29, 23, 7, -4, -2], referring to the table in Figure 3.

[0052] Hereinafter, when the filter coefficients obtained in this way are used in the horizontal direction, they will be written as fLH[p][i] (=fL[p][i]), and when they are used in the vertical direction, they will be written as fLV[p][i] (=fL[p][i]).

[0053] (Step A3) The reference image generated from the encoded frame to be referenced is resampled according to scalingRatio[x] so that it has the same resolution as the frame to be encoded. For example, the position of each pixel when the frame to be encoded is scaled by scalingRatio[x] is determined with 1 / 16 pixel accuracy. In addition, the reference image is interpolated by 16 times using the interpolation filter determined in step A2. For example, if scalingRatio[x] is 1.25 or more, a new sampling point (x3+p x ,y3+p y ) is calculated. Here, the coordinates of the pixel in question in the reference image are (x i ,y i ), the coordinates of the adjacent pixel on the left are (x (i-1) ,y i ), and the coordinates of the pixel adjacent below are (x i ,y (i-1)) notation is used. x , p y are integers modulo 16, and are indices of coordinates obtained by dividing the coordinates of adjacent pixels in the horizontal and vertical directions into 16 parts, respectively, and L(x, y) represents the luminance value of the coordinates (x, y). a is a normalization constant.

[0054]

number

[0055] Here, fLH[p][i] and fLV[p][i] are generated from the table in Figure 4. n are the coordinate values ​​from y0 to y7. The samples generated in this way are called the upsampled image. This upsampled image is thinned out to generate a resampled reference image with the same resolution as the frame to be coded from the reference image.

[0056] 6 and 7 are simplified diagrams for easily explaining the thinning method. The thinning method is the same for both the vertical and horizontal directions, so for simplicity, only the horizontal direction will be explained. It is assumed that both the reference image 601 and the encoding target frame 602 are composed of unit areas 603. It is assumed that one pixel value is defined for each unit area. In other words, the reference image 601 is a 6x6 pixel image made up of 6x6 unit areas, and the encoding target frame 602 is a 4x4 pixel image made up of 4x4 unit areas. The origin is the upper left vertex of the entire image, each unit area is 1x1 in size, and the coordinates of each pixel value are the coordinates of the upper left vertex of the unit area. The coordinate values ​​of each pixel in the encoding target frame are multiplied by scalingRatio[x] (3 / 2 in the example of FIG. 6). Then, the x coordinates of the encoding target frame 602 can be calculated as (0, 3 / 2, 6 / 2, 9 / 2) for (0, 1, 2, 3), and these are stored, for example, in H[x] of the coordinate array 703. Here, the image 604 is an image obtained by resampling the reference image 601 to have the same number of pixels as the encoding target frame 602. The pixel values ​​of each pixel block of the resampled image 604 may be constructed using pixel values ​​corresponding to the coordinates of the enlarged image from the upsampled image.

[0057] An example of the construction method will now be described with reference to Fig. 7. Reference numeral 701 denotes a unit area at y=0 (four unit areas in the top row) in the encoding target frame 602. Reference numeral 704 denotes a unit area at y=0 in the reference image 601 (six unit areas in the top row).

[0058] Here, the luminance value of the reference image 601 is Y[x], and the luminance value of the resampled image 604 is Y'[x]. Note that x is the x-coordinate position of the luminance value. Then, to obtain the luminance value of coordinate position x of the resampled image 604, H[x] in the coordinate array 703 is referenced, the coordinate position of the reference image 601 is found, and the luminance value of that coordinate position can be obtained. When limited to the explanation of the x-coordinate position, it can be written as follows:

[0059] Y'[x]=Y[H[x]] For example, since the x-coordinate position of the desired luminance value in unit area 702 is 1, x=1 is set, H[1]=3 / 2 is obtained from the coordinate array, and the value of Y[3 / 2] is adopted. Reference numeral 705 indicates the result of interpolating the unit area at coordinate position = 1 in unit area 704 where y=0 using the above filter into 16 areas. These "areas" are 1 / 16 the size of the unit area. Each area is assigned a pixel value calculated as a result of the filter. Then, Y[3 / 2] is obtained by taking the luminance value of the coordinate corresponding to 3 / 2 (=1+8 / 16) of unit area 706. In this way, a resampled reference image with the same resolution can be constructed by upsampling the reference image and thinning out pixel values ​​other than those corresponding to the enlarged pixel positions of the frame to be coded.

[0060] Inter prediction is a process of predicting pixels of a block to be coded by referencing pixels of an encoded frame, or, if the number of pixels of the encoded frame differs from the number of pixels of the frame to be coded, pixels of a resampled image constructed using the above method. For simplicity, the encoded frame and the resampled image are collectively referred to as the inter prediction target image. For example, if there is no motion between the reference encoded frame and the inter prediction target image, the pixels of the block to be coded are predicted using pixels at the same positions in the inter prediction target image. In such a case, a (0, 0) motion vector indicating no motion is included in the prediction information. On the other hand, if there is motion between frames for the block to be coded, the motion vector (MVx, MVy) is included in the prediction information.

[0061] 8 is a diagram illustrating a motion vector when a scaling window is set. The relationship between the motion vector, a block to be coded in a frame to be coded, and a block to be referenced in a reference image is shown in FIG.

[0062] Reference number 800 indicates a frame to be coded, reference number 801 indicates a block to be coded in the frame to be coded 800, and reference number 802 indicates a scaling window in the frame to be coded 800. When expressing coordinates, the upper left vertex of the frame to be coded 800 is set as the origin.

[0063] A current block 801 is a block to be coded using inter prediction, and the coordinates of its upper left vertex are (xSb, ySb). A scaling window 802 is a scaling window set for a current frame 800, and SWl_offset, one of its control parameters, is set to a.

[0064] Reference number 803 indicates the coded frame referenced by the coding target block 801, and includes pixels referenced by the luminance of coordinates in the coding target block 802. Note that when expressing coordinates, the upper left vertex of the coded frame 803 is set as the origin.

[0065] Reference numeral 804 denotes a reference area referenced by pixels included in the current block 801 to be coded, and the coordinates of its upper left vertex are (refxL, refyL). This vertex is the point at which the luminance value referenced by the luminance sample at the upper left vertex of the current block to be coded is calculated, and is not limited to a point with coordinates expressed as integers where an actual luminance sample of the reference frame exists. The luminance value of the coordinate expressed as a decimal may be calculated from the luminance value of the integer coordinate value using, for example, the aforementioned interpolation filter.

[0066] Reference numeral 805 denotes a scaling window set in the coded frame 803 that is referenced by the coding target block 801, and SWl_offset, one of its control parameters, is set to b.

[0067] Reference number 806 indicates a frame in which the encoding target frame 800 and the encoded frame 803 are superimposed, and is illustrated to show the relative positions of the encoding target block 801, reference area 804, scaling window 802, and scaling window 805.

[0068] A vector starting from the upper left vertex of the block to be coded 801 and ending at the upper left vertex of the reference area 804 is a motion vector 807. If the motion vector 807 is MV, it can be expressed as follows.

[0069] MV=(refxL-xSb, refxL-ySb) Furthermore, in VVC, the control values ​​a and b of the scaling windows 802 and 805 can be used to signal the following refined motion vector (hereinafter referred to as refMV) to the decoder side instead of the MV.

[0070] refMV=(refxL-xSb+ab, refxL-ySb) For simplicity's sake, we'll assume that there are no changes in size or in the y-axis direction between the scaling window of the frame to be coded and the scaling window set in the reference coded frame. If there is a change in the y-axis direction, the y component of refMV can be calculated using the scaling window control value, just like for a change in the x-axis direction. Furthermore, if there is a change in the size of the scaling window, the ScalingRatio can be calculated as follows, where ScalingRatio is the width of the scaling window set in the reference frame divided by the width of the scaling window set in the frame to be coded.

[0071] refMV=(refxL / ScailingRatio-xSb+ab / ScailingRatio, refxL / ScailingRatio-ySb) Here, for simplicity, it is assumed that the height of the scaling window does not change. However, even if the height of the scaling window changes, the same calculation can be performed by calculating the ScalengRatio in the height direction. When the movement direction and amount of the scaling window and the movement direction and amount of the moving object are the same or close to each other, the absolute value of refMV will be smaller than MV, and although there are various modes for signaling motion vectors, this contributes to overall improvement of coding efficiency. The motion vector (refMV) when the scaling window is set using the method described above can be calculated and signaled to the decoder side.

[0072] Returning to FIG. 1 , the operations of the transform / quantization unit 105, the inverse quantization / inverse transform unit 106, the image reproduction unit 107, and the in-loop filter unit 109 are as described above. The entropy coding unit 110 entropy codes, on a block-by-block basis, the quantization coefficients generated by the transform / quantization unit 105 and the prediction information generated by the prediction unit 104 to generate coded data. When coding the motion vector included in the prediction information, a difference between the motion vector of the block to be coded and the predicted motion vector calculated by the prediction unit 104, an identifier indicating which of the candidate predicted motion vectors has become the predicted motion vector, and the like are coded. The entropy coding method is not limited to a specific method, and Golomb coding, arithmetic coding, Huffman coding, and the like can be used.

[0073] The integrated encoding unit 111 encodes the image analysis information generated by the image analysis unit 102 and the scaling window control information generated by the generation unit 103 to generate a frame resolution information code and a scaling window information code, respectively. The encoding method may be, for example, Golomb encoding. The integrated encoding unit 111 then generates header code data including the frame resolution information code and the scaling window information code, and multiplexes the header code data with the code data generated by the entropy encoding unit 110 to generate a bitstream. The integrated encoding unit 111 then outputs the generated bitstream. The output destination of the bitstream is not limited to a specific output destination. For example, the integrated encoding unit 111 may output (store) the generated bitstream to the storage unit 906 or the memory unit 907, or may transmit it to the terminal device 1102 via the communication unit 908 over the network 1103.

[0074] Next, processing related to setting a scaling window in the image capturing device 1101 will be described with reference to the flowchart in Fig. 10. Note that the control unit 901 controls the processing speed of the entire loop in Fig. 10 so that captured images can be output at a specified frame rate.

[0075] In step S1001, the control unit 901 receives from the terminal device 1102 a "setting command for setting pan and tilt setting values" that has been set in the terminal device 1102, and stores the received setting command in the memory unit 907. Then, under the control of the control unit 901, the pan / tilt control unit 903 changes the current pan and tilt in accordance with the "pan and tilt setting values" in the setting command.

[0076] In step S1002, the image analysis unit 102 obtains the optical axis change value from the previous frame, and the scaling window control unit 904 calculates a scaling window control value according to the obtained optical axis change value. Specifically, the scaling window control unit 904 calculates the "amount of change in the number of samples in the horizontal direction" (amount of pan change) according to the following equation (9):

[0077] Change in the number of horizontal samples = (Width) x (tan(horizontal angle of view / 2) / tan(horizontal optical axis deviation angle / 2)) … (9) Although equation (9) is an equation for the pan direction, the tilt direction can be calculated in a similar manner. Next, the scaling window control unit 904 adds the "amount of change in the number of samples in the horizontal direction" calculated according to equation (9) to the scaling window control value of the previous frame, and calculates the result as the scaling window control value of the frame to be coded.

[0078] In step S1003, the scaling window control unit 904 determines whether the scaling window control value calculated in step S1002 exceeds a predetermined range. If the result of this determination is that the scaling window control value calculated in step S1002 exceeds the predetermined range, the process proceeds to step S1004. On the other hand, if the scaling window control value calculated in step S1002 does not exceed the predetermined range, the process proceeds to step S1005. This "predetermined range" will be described in detail later.

[0079] In step S1004, the scaling window control unit 904 recalculates the scaling window control value calculated in step S1002 so that it falls within a predetermined range. Details of the processing in step S1004 will be described later. In step S1005, the generation unit 103 stores the calculated scaling window control value in the PPS.

[0080] The process of calculating the scaling window control value according to equation (9) will be described using a specific example shown in Fig. 12. In particular, the behavior when the image capture device 1101 continues panning to the right and reaches the left limit of the scaling window control value in the VVC standard will be described.

[0081] Angle of view range 1201 indicates the angle of view range of the imaging device 1101 in frame 1, and angle of view range 1202 indicates the angle of view range of the imaging device 1101 in frame 2 following frame 1. Angle of view range 1203 indicates the angle of view range of the imaging device 1101 in frame 2' (an alternative case of frame 2) following frame 1, and angle of view range 1204 indicates the angle of view range of the imaging device 1101 in frame 3 following frame 2 (2'). The imaging device 1101 is panning to the right, as the angle of view range changes from angle of view range 1201 to angle of view range 1202 (angle of view range 1203) to angle of view range 1204. b in FIG. 12 indicates the angle of view panned to the right.

[0082] Scaling window 1205 is the scaling window in frame 1, and scaling window 1206 is the scaling window in frame 2. Furthermore, scaling window 1207 is the scaling window in frame 2', and scaling window 1208 is the scaling window in frame 3. "a" in FIG. 12 is SWl_offset.

[0083] Scaling window 1205 in frame 1 is set to scaling window 1206 at a position calculated using equation (9) in frame 2, and reaches the left end of the scaling window control value defined by the VVC standard. In such a case, the position of the scaling window is reset to the right end position of the scaling window control value defined by the VVC standard on the side in the direction of panning, as represented by scaling window 1207 in frame 2'. Here, the position of the scaling window does not necessarily have to be set to the right end, and it may be at the center of the angle of view range, for example.

[0084] Moving the scaling window in this way, as in scaling window 1207, signals an incorrect vector offset value, resulting in reduced coding efficiency in inter prediction. This timing for resetting the scaling window provides a good opportunity to insert an IDR frame that does not need to reference the previous frame. In frame 3, panning further to the right occurs, and scaling window 1208 is reset using equation (9).

[0085] The process of calculating the scaling window control value according to equation (9) will be described using a specific example shown in Fig. 13. In particular, the behavior will be described when the imaging device 1101 continues panning to the right and the reference region 804 described in Fig. 8 falls outside the inter prediction range.

[0086] Angle of view range 1301 indicates the angle of view range of the imaging device 1101 in frame 1, and angle of view range 1302 indicates the angle of view range of the imaging device 1101 in frame 2 following frame 1. Angle of view range 1303 indicates the angle of view range of the imaging device 1101 in frame 2' (an alternative case of frame 2) following frame 1, and angle of view range 1304 indicates the angle of view range of the imaging device 1101 in frame 3 following frame 2 (2'). The imaging device 1101 is panning to the right, as the angle of view range changes from angle of view range 1301 to angle of view range 1302 (angle of view range 1303) to angle of view range 1304. b in FIG. 13 indicates the angle of view panning to the right.

[0087] Scaling window 1305 is the scaling window for frame 1, and scaling window 1306 is the scaling window for frame 2. Furthermore, scaling window 1307 is the scaling window for frame 2', and scaling window 1308 is the scaling window for frame 3. "a" in FIG. 13 is SWl_offset. Furthermore, in FIG. 13, a coding target block 801 is set at the center of the person in frame 2, and a reference region 804 is set at the center of the person in frame 1.

[0088] Reference numeral 1311 indicates the inter prediction range in frame 1 corresponding to the current block 801 to be coded in frame 2. Furthermore, panning to the right has caused the reference area 804 to fall outside the inter prediction range 1311. This indicates that inter prediction does not work, but intra prediction does work.

[0089] The timing when inter prediction is no longer effective is also a good opportunity to insert an IDR frame and reset the scaling window. In such a case, as represented by the scaling window 1307 of frame 2', the position of the scaling window is reset to the right edge of the scaling window control value defined by the VVC standard in the panning direction. Here, the position of the scaling window does not necessarily have to be set to the right edge, and may be the center position of the angle of view range, for example. In frame 3, panning is further to the right, and the scaling window 1308 is reset using equation (9).

[0090] The process of calculating the scaling window control value according to equation (9) will be described using a specific example shown in Fig. 14. In particular, the behavior when the image capture device 1101 continues panning to the right and the range limit of the scaling window control value in the VVC standard is not taken into consideration will be described.

[0091] Angle of view range 1401 indicates the angle of view range of the imaging device 1101 in frame 1, and angle of view range 1402 indicates the angle of view range of the imaging device 1101 in frame N, which is several frames after frame 1. Angle of view range 1403 indicates the angle of view range of the imaging device 1101 in frame M, which is several frames after frame N. The imaging device 1101 is panning to the right, as the angle of view range changes from angle of view range 1401 to angle of view range 1402 to angle of view range 1403.

[0092] Scaling window 1404 is the scaling window in frame 1. a1 is SWl_offset in frame 1. aN is SWl_offset seen from frame N, and SWl_offset is set beyond the range limit of the scaling window control value in the VVC standard. In this case, since the scaling window is reset using equation (9), there are advantages in that it is not necessary to reset the scaling window or insert an IDR frame while still benefiting from the improvement in coding efficiency described in FIG. 8.

[0093] On the other hand, for example, if you continue to pan in one direction using a swivel, or if you continue to pan while zoomed to the telephoto end even if you are not using a swivel, the absolute value of the scaling window control value will become large, and there is a concern that coding efficiency will decrease.

[0094] In such cases, such concerns can be avoided by resetting the scaling window at the appropriate timing and inserting the IDR frame. In the case of a rotating aircraft, the scaling window may be reset and the IDR frame may be inserted when it has completed one rotation (2π). Alternatively, the scaling window may be reset at the timing described later in Figure 15.

[0095] The process of calculating the scaling window control value according to equation (9) will be described using a specific example shown in Fig. 15. In particular, the behavior when the image capture device 1101 continues panning to the right and the range limit of the scaling window control value in the VVC standard is not taken into consideration will be described.

[0096] Angle of view range 1501 indicates the angle of view range of the imaging device 1101 in frame 1, and angle of view range 1502 indicates the angle of view range of the imaging device 1101 in frame N, which is several frames after frame 1. Angle of view range 1503 indicates the angle of view range of the imaging device 1101 in frame N' (an alternative case of frame N), which is several frames after frame 1. Angle of view range 1504 indicates the angle of view range of the imaging device 1101 in frame M, which is several frames after frame N (N'). The imaging device 1101 is panning to the right, as the angle of view range changes from angle of view range 1501 to angle of view range 1502 (1503) to angle of view range 1504.

[0097] Scaling window 1505 is the scaling window in frame 1, scaling window 1506 is the scaling window in frame N, scaling window 1507 is the scaling window in frame N′, and scaling window 1508 is the scaling window in frame M.

[0098] The scaling window control value of the scaling window 1506 reaches a position to the left (opposite the panning direction) by the width of the angle of view range 1502. At this point, the position of the scaling window is reset and an IDR frame is inserted. The scaling window control value of the scaling window 1507 is reset to a position to the right (in the panning direction) by the width of the angle of view range 1503.

[0099] The process of calculating the scaling window control value according to equation (9) will be described using a specific example shown in Fig. 16. In particular, the behavior when the image capture device 1101 continues panning to the right and reaches the left limit of the scaling window control value in the VVC standard will be described.

[0100] Angle of view range 1601 indicates the angle of view range of the imaging device 1101 in frame 1, and angle of view range 1602 indicates the angle of view range of the imaging device 1101 in frame 2 that follows frame 1. Angle of view range 1603 indicates the angle of view range of the imaging device 1101 in frame 3 that follows frame 2, and angle of view range 1604 indicates the angle of view range of the imaging device 1101 in frame 4 that follows frame 3. The imaging device 1101 is panning to the right, as the angle of view range changes from angle of view range 1601 to angle of view range 1602 to angle of view range 1603 to angle of view range 1604. b in FIG. 16 indicates the angle of view panned to the right.

[0101] A scaling window 1605 is a scaling window in frame 1, and a scaling window 1606 is a scaling window in frame 2. Furthermore, a scaling window 1607 is a scaling window in frame 3, and a scaling window 1608 is a scaling window in frame 4. In FIG. 16, a is SWl_offset.

[0102] The scaling window 1605 in frame 1 is set to a scaling window 1606 at a position calculated using equation (9) in frame 2, and reaches the left end of the scaling window control value defined by the VVC standard. In such a case, the position of the scaling window is reset to the right end position of the scaling window control value defined by the VVC standard on the side in the direction of panning, as represented by scaling window 1607 in frame 3. Here, the position of the scaling window does not necessarily have to be set to the right end, and it may be at the center position of the angle of view range, for example.

[0103] At this time, unlike the method described in Fig. 12, frame 3 is a skip frame. Here, a skip frame is a frame that is output when a frame with no difference from the previous frame is inter-predicted, and coding efficiency does not decrease even if the frame is moved as in scaling window 1607. Next, in frame 3, panning is further performed to the right, and the position of scaling window 1608 is reset using equation (9).

[0104] Although the video will skip at frame 3, it is possible to transmit frame 4 without skipping a frame by inserting frame 3 at, for example, half the normal frame interval. The insertion of various frames in the above description is a process performed by the encoding unit 905 during encoding.

[0105] The overall operation of the imaging device 1101 will be described with reference to the flowchart of Fig. 17. Note that the processing in each step in Fig. 17 is as described above, and therefore detailed description will be omitted.

[0106] In step S1701, the imaging unit 902 captures moving images and generates captured images for each frame. In step S1702, the captured images are subjected to the above-mentioned processing related to setting the scaling window (including processing according to the flowchart in FIG. 10) and encoding processing by the encoding unit 905. In step S1703, the encoding unit 905 outputs the bitstream generated by the processing in step S1702.

[0107] In this manner, in this embodiment, the position of the scaling window that signals the center of the reference image used for inter-prediction in encoding the captured image is controlled according to the amount of movement of the capturing direction of the imaging device when the capturing direction is controlled. This allows, for example, in an imaging device that can control pan / tilt, to effectively use the scaling window so that the amount of code for the captured image does not increase.

[0108] [Second embodiment] In the first embodiment, the imaging device 1101 is described as changing the setting values ​​such as the shooting direction, zoom, and focus in response to instructions from the terminal device 1102, but it may also be configured to recognize the tracking subject in the captured image and change the setting values ​​in response to the results of the recognition.

[0109] Furthermore, in the first embodiment, several conditions are listed as conditions that must be met in order to reset the position of the scaling window, but the above conditions are merely examples and are not limited to specific conditions.

[0110] The numerical values, processing timing, processing order, processing subject, data (information) configuration / acquisition method / sending destination / sending source / storage location, etc. used in the above embodiment are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0111] In addition, some or all of the above-described embodiments may be used in appropriate combination, and some or all of the above-described embodiments may be selectively used.

[0112] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0113] The invention of this specification includes the following imaging device and imaging device control method. (Item 1) An imaging device, encoding means for encoding the captured image; a control means for controlling a position of a scaling window that signals a center of a reference image used in inter prediction in the encoding, in accordance with a movement amount of the shooting direction of the imaging device when the shooting direction is controlled; An imaging device comprising: (Item 2) 2. The imaging device according to item 1, wherein the control means resets the position of the scaling window in the direction of progress of the imaging direction when controlling the imaging direction. (Item 3) 3. The imaging device according to item 1 or 2, wherein the control means resets the position of the scaling window when a specified condition is satisfied. (Item 4) 4. The imaging device according to item 3, wherein the control means resets the position of the scaling window when the control value of the scaling window exceeds a range defined by the VVC standard. (Item 5) 5. The imaging device according to any one of items 1 to 4, wherein the encoding means inserts an IDR frame when the position of the scaling window is reset. (Item 6) 5. The imaging device according to any one of items 1 to 4, wherein the encoding means inserts a skip frame when the position of the scaling window is reset. (Item 7) moreover, 7. The imaging device according to any one of items 1 to 6, further comprising a means for controlling the imaging direction in response to a command received from an external device. (Item 8) moreover, 8. The imaging device according to any one of items 1 to 7, further comprising a means for outputting the result of encoding by the encoding means to an external device. (Item 9) 9. The imaging device according to any one of items 1 to 8, wherein the encoding is VVC. (Item 10) A control method for an imaging device, comprising: an encoding step in which an encoding means of the imaging device encodes a captured image; a control step in which a control means of the imaging device controls a position of a scaling window that signals a center of a reference image to be used for inter-prediction in the encoding, in accordance with an amount of movement of the imaging direction when the imaging direction of the imaging device is controlled; 11. A method for controlling an imaging device, comprising:

[0114] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0115] 901: Control unit 902: Imaging unit 903: Pan / tilt control unit 904: Scaling window control unit 905: Encoding unit 906: Storage unit 907: Memory unit 908: Communication unit

Claims

1. An imaging device, encoding means for encoding the captured image; a control means for controlling a position of a scaling window that signals a center of a reference image used in inter prediction in the encoding, in accordance with a movement amount of the shooting direction of the imaging device when the shooting direction is controlled; An imaging device comprising:

2. 2. The imaging device according to claim 1, wherein the control means resets the position of the scaling window in the direction of progress of the imaging direction when controlling the imaging direction.

3. 2. The imaging apparatus according to claim 1, wherein said control means resets the position of said scaling window when a specified condition is satisfied.

4. 4. The imaging device according to claim 3, wherein said control means resets the position of said scaling window when a control value of said scaling window exceeds a range defined by a VVC standard.

5. 2. The imaging device according to claim 1, wherein said encoding means inserts an IDR frame when the position of said scaling window is reset.

6. 2. The imaging device according to claim 1, wherein said encoding means inserts a skip frame when the position of said scaling window is reset.

7. moreover, 2. The imaging device according to claim 1, further comprising a means for controlling the imaging direction in response to a command received from an external device.

8. moreover, 2. The imaging apparatus according to claim 1, further comprising: means for outputting the result of encoding by said encoding means to an external device.

9. 2. The imaging device according to claim 1, wherein the encoding is VVC.

10. A control method for an imaging device, comprising: an encoding step in which an encoding means of the imaging device encodes a captured image; a control step in which a control means of the imaging device controls a position of a scaling window that signals a center of a reference image to be used for inter-prediction in the encoding, in accordance with an amount of movement of the imaging direction when the imaging direction of the imaging device is controlled; 11. A method for controlling an imaging device, comprising:

Images