Imaging apparatus, and method for controlling imaging apparatus

Abstract

To provide a technique to allow an improvement of the quality of moving images and a reduction in the amount of codes even when the angle of view is changed.SOLUTION: An imaging apparatus picks up moving images to generate a frame, and performs encoding by using the generated frame. When instructed to change the angle of view, the imaging apparatus determines whether to generate a high-resolution frame. In the encoding, when determining to generate the high-resolution frame, the imaging apparatus performs the encoding by using the high-resolution frame as a long-term reference frame.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to encoding techniques. [Background technology]

[0002] In recent years, with the expansion of the video distribution market, there has been an increase in filming systems for filming events such as weddings and lectures using cameras capable of remote control and video recording over a network. While remotely controlled cameras are generally operated using a hardware controller, they can also be operated using an application (software controller) that runs on a PC or mobile device. Remote control includes PTZ operation. The application can also receive and play back video images captured and distributed by the camera.

[0003] Cameras encode video images for efficient distribution over networks and efficient storage. While H.264 and H.265 are well-known encoding methods, new encoding methods have been proposed based on various new requirements. It is known that encoding efficiency drops particularly during panning, tilting, and zooming, but encoding methods that do not drop in encoding efficiency even in these situations have also been proposed.

[0004] The Versatile Video Coding (VVC) coding method (hereinafter referred to as VVC) is known as a coding method for compressing and recording moving images. VVC introduces a technology called Reference Picture Resampling (RPR) (hereinafter referred to as RPR) to improve coding efficiency. RPR is a technology that allows an image with a different resolution from the image to be decoded to be used as a reference image, making it possible to change the resolution even in the case of inter-frame compression.

[0005] Patent Document 1 discloses a technology in which, in a camera without pan, tilt, or zoom mechanisms, an image is enlarged and displayed on the display in response to a digital zoom operation, and while the image is being enlarged, a higher resolution image is requested, and once the image acquisition is complete, the image currently being displayed is replaced with the new image. This makes it possible to display an image with a higher resolution than with a typical digital zoom.

[0006] Furthermore, Patent Document 2 discloses a technology that automatically changes the application level of high-resolution processing according to the movement speed of a zoom lens and a focus lens. For example, the application level of high-resolution processing begins to increase only when the lens movement speed becomes slow enough that the effect of high-resolution processing can be recognized with the naked eye, and is controlled so that the effect is maximized when the subject is stationary. Conversely, if high-resolution processing is performed while the lens is moving at high speed, the application level of high-resolution processing is reduced or disabled if it has an adverse effect, such as increasing high-frequency components in the subsequent compression process and working against information volume compression.

[0007] Furthermore, Patent Document 3 discloses a technology for displacing an image sensor in a frame cycle by completing a series of operations, including displacement, exposure, and image signal readout, within one frame, thereby making it possible to capture high-resolution images with a number of pixels greater than the number of pixels in the image sensor. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-189503 [Patent Document 2] Japanese Patent Application Laid-Open No. 2014-216692 [Patent Document 3] Japanese Patent Application Laid-Open No. 2004-15341 Summary of the Invention [Problem to be solved by the invention]

[0009] However, the technology disclosed in the above-mentioned Patent Document 1 does not take into consideration cameras with pan, tilt, and zoom mechanisms. The technology disclosed in Patent Document 2 does not consider the decrease in compression efficiency that occurs when the angle of view changes during zooming in, increasing the difference between frames, and is therefore unable to reduce the amount of coding while maintaining video quality. The technology disclosed in Patent Document 3 discloses a method for acquiring still images, but does not consider moving images. The present invention provides a technology that enables the improvement of the image quality of moving images and the reduction of the amount of coding, even when the angle of view is changed. [Means for solving the problem]

[0010] One aspect of the present invention includes a generation means for capturing moving images and generating frames, an encoding means for performing encoding using the frames generated by the generation means, and a determination means for determining, when an instruction to change the angle of view is received, whether or not to cause the generation means, which is generating frames at a first resolution, to generate frames at a second resolution higher than the first resolution, and when it is determined that the generation means should generate frames at the second resolution, the encoding means performs encoding using the frames at the second resolution generated by the generation means as long-term reference frames. [Effects of the Invention]

[0011] According to the configuration of the present invention, it is possible to improve the image quality of moving images and reduce the amount of code even when the angle of view is changed. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 9 is a block diagram showing an example of the configuration of an encoding unit 905. [Figure 2] A diagram of how to set the scaling window. [Figure 3] FIG. 10 is a diagram showing an example of a table configuration. [Figure 4] FIG. 10 is a diagram showing an example of a table configuration. [Figure 5] FIG. 10 is a diagram showing an example of a table configuration. [Figure 6] A simplified diagram to easily explain the thinning method. [Figure 7] A simplified diagram to easily explain the thinning method. [Figure 8] FIG. 1 is a diagram showing an example of a system configuration. [Figure 9] FIG. 8 is a block diagram showing an example of the hardware configuration of an imaging device 801. [Figure 10] 8 is a flowchart of an encoding process performed by the imaging device 801. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0014] [First embodiment] First, an example of the configuration of a system according to this embodiment will be described with reference to Fig. 8. As shown in Fig. 8, the system according to this embodiment includes an imaging device 801 such as a network camera or an IP streaming camera, and terminal devices 802a and 802b such as PCs and tablet terminals. The imaging device 801 and the terminal devices 802a and 802b are connected to a network 803, which allows data communication between them. Hereinafter, the terminal devices 802a and 802b will be collectively referred to as terminal devices 802.

[0015] The imaging device 801 is a device that captures moving images and generates frames (captured images). In response to commands received from the terminal device 802, the imaging device 801 transmits frames and various information to the terminal device 802, and changes the pan, tilt, zoom, focus, etc. of the imaging device 801. Note that the imaging device 801 may transmit frames and various information to the terminal device 802 regardless of whether it has received a command.

[0016] The terminal device 802 generates various commands in response to user operations and transmits the generated commands to the imaging device 801. The terminal device 802 can then display or store frames and various information received from the imaging device 801 in response to the transmission.

[0017] The network 803 is realized by, for example, a plurality of routers, switches, cables, etc. that comply with a communication standard such as ETHERNET (registered trademark). Note that the network 803 may also be realized by the Internet, a wired LAN (Local Area Network), a wireless LAN, a WAN (Wide Area Network), etc.

[0018] Next, an example of the hardware configuration of the imaging device 801 will be described using the block diagram in Fig. 9. The control unit 901 executes various processes using computer programs and data stored in the memory unit 907. In this way, the control unit 901 controls the overall operation of the imaging device 801, and also executes or controls various processes that will be described as processes performed by the imaging device 801.

[0019] For example, the control unit 901 analyzes a camera control command transmitted from the terminal device 802 and received by the communication unit 908 via the network 803, and performs processing according to the results of the analysis. Here, the camera control commands can be roughly classified into request commands for requesting acquisition of moving images or setting values ​​of the imaging device 801, and setting commands for requesting the imaging device 801 to set setting values.

[0020] When the control unit 901 receives a "request command for requesting acquisition of a video image" from the terminal device 802, the control unit 901 transmits the bit stream generated by the encoding unit 905 to the terminal device 802 via the communication unit 908 and the network 803. The control unit 901 can also store the bit stream in the storage unit 906.

[0021] Also, assume that the control unit 901 receives a "request command requesting acquisition of the zoom, focus, pan, and tilt setting values ​​of the imaging device 801" from the terminal device 802. At this time, the control unit 901 acquires the "zoom and focus setting values" and the "pan and tilt (imaging direction) setting values," and transmits the acquired setting values ​​to the terminal device 802 via the network 803 using the communication unit 908. Here, the setting values ​​of zoom, focus, pan, and tilt include not only the current values, but also related information such as the range of values ​​that can be set in the imaging device 801.

[0022] The method of acquiring the "zoom and focus setting values" and the "pan and tilt (shooting direction) setting values" is not limited to a specific method. For example, the "zoom and focus setting values" may be acquired from the angle of view control unit 903. Also, for example, the "pan and tilt (shooting direction) setting values" may be acquired from an imaging attitude control unit (not shown) that controls the shooting attitude of the imaging device 801.

[0023] Also, it is assumed that the control unit 901 receives a "setting command to set the zoom, focus, pan, and tilt setting values" from the terminal device 802. At this time, the angle of view control unit 903, under the control of the control unit 901, changes the angle of view by changing the current zoom and focus in accordance with the "zoom and focus setting values" in the setting command. Similarly, the shooting attitude control unit, under the control of the control unit 901, changes the current pan and tilt in accordance with the "pan and tilt setting values" in the setting command. As a result, the setting values ​​for zoom, focus, pan, and tilt set by the terminal device 802 are reflected in the imaging device 801. Note that detailed description of focus, pan, and tilt will be omitted.

[0024] The imaging unit 902 includes an optical system including multiple lenses, an imaging element, respective drive control circuits, and various image processing circuits. The imaging element outputs an analog image signal corresponding to an optical image received through the optical system, and the image processing circuit performs various processes on the analog image signal, such as A / D conversion, development, various color processing, and resolution conversion, to generate a frame.

[0025] The angle of view control unit 903 has drive systems and drive motors for the focus lens and zoom lens in the imaging unit 902. For example, the angle of view control unit 903 controls the position of the zoom lens in the imaging unit 902 based on a "zoom setting value" under the control of the control unit 901. Here, the zoom setting value is managed, for example, as the focal length.

[0026] When an instruction to change the angle of view of the imaging device 801 is given, the generation determination unit 904 determines whether or not to cause the imaging unit 902, which is generating frames at a first resolution, to generate frames at a second resolution that is higher than the first resolution.

[0027] The encoding unit 905 performs encoding using each frame generated by the imaging unit 902, and generates a bitstream including the results of the encoding. Details of the encoding unit 905 will be described later.

[0028] The storage unit 906 is a non-volatile memory device that functions as internal storage of the imaging device 801 and / or external storage of the imaging device 801. For example, a memory device such as an HDD, SSD, or FROM can be used as the internal storage of the imaging device 801 or the external storage of the imaging device 801. The storage unit 906 stores computer programs and data for causing the control unit 901 to execute or control the overall operation of the imaging device 801. The computer programs and data stored in the storage unit 906 are loaded into the memory unit 907 as appropriate under the control of the control unit 901, and become targets for processing by the control unit 901.

[0029] The memory unit 907 is a volatile memory device such as a RAM. The memory unit 907 has an area for storing computer programs and data loaded from the storage unit 906, and an area for storing information such as commands received from the outside via the communication unit 908. The memory unit 907 also has an area for storing frames generated by the imaging unit 902. The memory unit 907 also has a work area used by the control unit 901 and the encoding unit 905 when performing various processes. In this way, the memory unit 907 can provide various areas as needed.

[0030] The communication unit 908 performs data communication with the terminal device 802 via the network 803. For example, the communication unit 908 transmits a bit stream generated by the imaging device 801 to the terminal device 802 via the network 803. Also, for example, the communication unit 908 receives a command transmitted from the terminal device 802 via the network 803, and transmits a response generated by the imaging device 801 in response to the command to the terminal device 802 via the network 803.

[0031] Next, the encoding process by the imaging device 801 will be described with reference to the flowchart in Fig. 10. Note that the control unit 901 controls the processing speed of the entire loop in Fig. 10 so that frames can be output at a specified frame rate.

[0032] In step S1001, the control unit 901 performs control in accordance with the command received from the terminal device 802. For example, when the control unit 901 receives from the terminal device 802 a "setting command for setting zoom and focus setting values" for zooming in, the control unit 901 generates a zoom-in instruction including the "zoom and focus setting values" included in the setting command. The control unit 901 then outputs the generated zoom-in instruction to the angle-of-view control unit 903. Upon receiving the zoom-in instruction, the angle-of-view control unit 903 changes the current zoom lens position and focus lens position to positions according to the "zoom and focus setting values" included in the zoom-in instruction, thereby changing the angle of view.

[0033] In step S1002, the generation determination unit 904 determines whether or not the control unit 901 output a zoom-in instruction to the angle of view control unit 903 in step S1001. If the result of this determination is that the control unit 901 output a zoom-in instruction to the angle of view control unit 903 in step S1001, the process proceeds to step S1004. On the other hand, if the control unit 901 did not output a zoom-in instruction to the angle of view control unit 903 in step S1001, the process proceeds to step S1003.

[0034] In step S1003, the imaging unit 902 captures a moving image to generate a frame (an image having a number of pixels equal to or less than the number of pixels of the imaging element of the imaging unit 902), and the control unit 901 stores the frame in the memory unit 907. In step S1003, the frame is stored in the memory unit 907 without being encoded.

[0035] In step S1004, the generation determination unit 904 determines whether or not a generation condition is satisfied for causing the imaging device 801 to generate a high-resolution frame (high-resolution frame). Various conditions can be applied as the generation condition, and it is not limited to a specific condition.

[0036] For example, the generation determination unit 904 acquires the latest frame (latest frame) and the frame one frame before (past) the latest frame (past frame) from the memory unit 907 as reference frames. The generation determination unit 904 then determines that the generation condition is satisfied if the difference calculated between the reference frames is less than a threshold, and determines that the generation condition is not satisfied if the difference is equal to or greater than the threshold. Possible cases in which the difference calculated between the reference frames is large include when the subject moves significantly between frames or when the image capture device 801 pans and tilts significantly. Since methods for calculating such differences are well known, detailed explanations thereof will be omitted. The number of frames acquired as reference frames is not limited to two, but may be three or more. When three or more frames are acquired as reference frames, the generation condition is determined to be satisfied if the cumulative difference between adjacent reference frames is less than a threshold, and the generation condition is determined to be not satisfied if the difference is equal to or greater than the threshold.

[0037] Also, for example, if the current zoom setting value is greater than the threshold value, the generation determination unit 904 may determine again that the generation condition is satisfied when the setting value exceeds a specified limit value.

[0038] If the generation condition is satisfied as a result of such determination, the process proceeds to step S1005, and if the generation condition is not satisfied, the process proceeds to step S1003.

[0039] In step S1005, the encoding unit 905 generates metadata describing the settings for setting a scaling window for a frame before zoom-in begins, and stores the generated metadata in the memory unit 907. The control unit 901 also generates metadata describing the settings for using a high-resolution frame as a long-term reference frame, and stores the generated metadata in the memory unit 907. Here, the long-term reference frame is, for example, a long-term reference picture in the HEVC standard or the VVC standard.

[0040] In step S1006, the generation determination unit 904 instructs the imaging unit 902 to generate a high-resolution frame. In step S1007, upon receiving the instruction to generate a high-resolution frame from the generation determination unit 904, the imaging unit 902 generates a high-resolution frame, which is an image with a number of pixels greater than the number of pixels of the image sensor of the imaging unit 902, using, for example, the technology disclosed in Patent Document 3. At this time, the magnification of the high-resolution frame generated by the imaging unit 902 can be changed depending on the zoom setting value. Note that the technology disclosed in Patent Document 3 is not the only option, and high-resolution frames may also be generated by other methods. That is, in step S1007, the imaging unit 902 generates a frame with a higher resolution than the resolution of the frame generated in step S1003 as the high-resolution frame.

[0041] The high-resolution frames may be generated by a functional unit other than the imaging unit 902, for example, the control unit 901. This also applies to the generation of normal frames that are not high-resolution frames.

[0042] In step S1008, the encoding unit 905 generates a bitstream by encoding the frame generated in step S1003 and the high-resolution frame generated in step S1007. In doing so, the encoding unit 905 references the metadata stored in the memory unit 907 in step S1005, sets a scaling window for the high-resolution frame, and performs encoding using the high-resolution frame as a long-term reference frame.

[0043] In step S1009, the control unit 901 transmits the bit stream generated in step S1008 to the terminal device 802 via the network 803 using the communication unit 908.

[0044] Next, an example configuration of the encoding unit 905 will be described using the block diagram in Fig. 1. The functional units shown in Fig. 1 may be implemented by hardware or software (computer program). In the latter case, the software is executed by the control unit 901, thereby realizing the functions of the corresponding functional units. The configuration shown in Fig. 1 may also be implemented by a combination of hardware and software. The encoding unit 905 acquires a frame to be encoded as an input image, and performs VVC encoding on the input image to generate a bitstream.

[0045] The image analysis unit 102 acquires a frame to be coded as an input image, analyzes the angle of view change value and the optical axis change value of the input image, and generates the analysis results as image analysis information. The image analysis unit 102 also divides the input image into multiple tile images based on the image characteristics of the input image and external inputs.

[0046] The generation unit 103 generates scaling window control information (RPR control information) including information on the scaling ratio and offset position of the motion vector required for decoding using RPR. The scaling window is a technology standardized by VVC, and is used for efficient coding by changing the scaling window in synchronization with changes in the angle of view when the zoom is changed, etc.

[0047] The prediction unit 104 performs prediction processing such as intra-prediction, which is intra-frame prediction, or inter-prediction, which is inter-frame prediction, on the tile images to generate predicted images, and calculates prediction errors from the tile images and the predicted images. The prediction unit 104 also generates information necessary for the prediction processing, such as prediction modes and motion vectors, as prediction information.

[0048] The transform / quantization unit 105 performs orthogonal transform on the prediction errors in units of blocks to generate transform coefficients, and quantizes the transform coefficients to generate quantized coefficients. The inverse quantization / inverse transform unit 106 inverse quantizes the quantized coefficients generated by the transform / quantization unit 105 to regenerate transform coefficients, and further performs inverse orthogonal transform on the regenerated transform coefficients to regenerate the prediction errors.

[0049] The frame memory 108 is a memory for storing a reproduced image (reproduced image). The image reproduction unit 107 generates a predicted image by appropriately referring to the frame memory 108 based on the prediction information generated by the prediction unit 104, generates a reproduced image from the generated predicted image and a prediction error, and stores the reproduced image in the frame memory 108.

[0050] The in-loop filter unit 109 generates a filtered reconstructed image by performing in-loop filtering such as deblocking filtering and sample adaptive offset on the reconstructed image stored in the frame memory 108. The in-loop filter unit 109 then stores the generated filtered reconstructed image in the frame memory 108.

[0051] The entropy coding unit 110 encodes the quantization coefficients generated by the transform / quantization unit 105 and the prediction information generated by the prediction unit 104 to generate coded data.

[0052] The integrated coding unit 111 generates header code data by encoding the image analysis information generated by the image analysis unit 102 and the scaling window control information generated by the generation unit 103. The integrated coding unit 111 then generates a bit stream including the header code data and the code data generated by the entropy coding unit 110, and outputs the generated bit stream.

[0053] Next, the operation performed by the encoding unit 905 having such a configuration to encode one frame of input image (hereinafter, sometimes simply referred to as a frame) will be described in more detail.

[0054] The image analysis unit 102 calculates the angle of view change value and optical axis change value of the frame. When an arbitrary frame is set as a reference frame, the angle of view change value may be the ratio of the angle of view of the reference frame to the frame to be encoded. Furthermore, when the center of the reference frame is set as the optical axis, the optical axis change value is a numerical value that represents the deviation of the image between the frame to be encoded and the reference frame. For example, PTZ information may be input as an external input, and the angle of view change value and optical axis change value may be calculated.

[0055] For example, the image analysis unit 102 may generate reduced frames by changing the degree of scaling of a reference frame (which may be a portion of the reference frame), calculate a correlation value between each reduced frame and the input frame, and use the degree of scaling that produces the largest correlation value as the angle of view change value.

[0056] In addition, the image analysis unit 102 may cut out the vicinity of the center of the reference image, calculate the correlation value with the image cut out using multiple coordinates as center coordinates within a certain range from the center of the input frame, and use the center coordinate of the cut-out with the largest correlation value as the optical axis change value.

[0057] The image analysis unit 102 may also accept metadata associated with the input frame and process the input frame, for example, to set a scaling window or a long-term reference frame.

[0058] The generation unit 103 signals the use of RPR by setting sps_ref_pic_resampling_enabled_flag in the Sequence Parameter Set (hereinafter, SPS) to 1. The generation unit 103 also stores the number of horizontal and vertical luminance pixels of the frame as pps_pic_width_in_luma_samples and pps_pic_height_in_luma_samples in the Picture Parameter Set (hereinafter, PPS), respectively.

[0059] The generation unit 103 calculates a scaling window control value consisting of the following as scaling window control information from the angle of view change value and the optical axis change value obtained by the image analysis unit 102:

[0060] The scaling window control values ​​are made up of pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, and pps_scaling_win_bottom_offset, and can be calculated as follows.

[0061] For example, let SW_w and SW_h be the number of samples of horizontal and vertical luminance values ​​in the scaling window of the reference frame, respectively, and let Px be the horizontal deviation of the optical axis change value and Py be the vertical deviation. In this case, pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, and pps_scaling_win_bottom_offset can be calculated based on the following equations (1-1), (1-2), (1-3), and (1-4), respectively.

[0062] pps_scaling_win_left_offset=C*((pps_pic_width_in_luma_samples-SW_w) / 2+Px) … (1-1) pps_scaling_win_right_offset=C*((pps_pic_width_in_luma_samples-SW_w) / 2-Px) … (1-2) pps_scaling_win_top_offset=C*((pps_pic_height_in_luma_samples-SW_h) / 2+Py) … (1-3) pps_scaling_win_bottom_offset=C*((pps_pic_height_in_luma_samples-SW_h) / 2-Py) … (1-4) Here, C is the reciprocal of the subsampling rate, which is 1 / 2 for 4:2:0, for example. Furthermore, if the reference frame is the previous frame and there is no scaling window, SW_w and SW_h may be calculated as follows, with Rh representing the change in the angle of view on the vertical axis and Rw representing the change in the angle of view on the horizontal axis:

[0063] SW_w=Rw*pps_pic_width_in_luma_samples … (1-5) SW_h=Rh*pps_pic_height_in_luma_samples … (1-6) Here, a method for setting a scaling window according to this embodiment will be described with reference to FIG. 2. In this embodiment, fluctuations in the angle of view and changes in resolution occur simultaneously during zooming. Therefore, the reference frame used to set the scaling window is set to the frame after zooming, and setting of the scaling window is started in a frame (e.g., an IDR frame) that does not use RPR before zooming begins. In FIG. 2, reference numbers 201 and 202 indicate frames before zooming begins, reference numbers 203 and 204 indicate frames during zooming, and reference number 205 indicates a frame after zooming ends.

[0064] When the imaging device 801 receives a zoom operation, it generates a frame 201 that does not use RPR and sets a scaling window 206 for the frame 201. The scaling window 206 is the range that is captured in a frame 205 after the zoom operation is completed, calculated from the angle of view after the zoom operation.

[0065] Furthermore, the imaging device 801 generates a high-resolution frame 202 that references frame 201 using RPR as a frame before ZOOM begins, and sets a scaling window 207 for this high-resolution frame 202 in the same manner as the scaling window 206. Here, the resolution of frame 202 may be determined so that the resolution of scaling window 207 is the same as the resolution of frame 205 after ZOOM ends. After ZOOM begins, frames may be generated at the same resolution as frame 201. For example, the size of scaling window 208 for frame 203 is larger than that of scaling window 206 because ZOOM has started. Frame 204 is also generated at the same resolution as frame 201, and similarly, the scaling window 209 for frame 204 is larger than that of scaling window 208. After ZOOM ends, frame 205 is generated at the same resolution as the resolution within scaling window 207, and since the scaling window covers the entire image, no setting is required. In this way, frames that do not use RPR, which requires a large amount of coding to generate a scaling window, can be generated at a lower resolution.

[0066] 1, the prediction unit 104 divides the tile images divided by the image analysis unit 102 into multiple blocks, and performs prediction processing on a block-by-block basis to generate prediction errors and prediction information. Here, the prediction processing performed by the prediction unit 104 and the prediction information generated by the prediction unit 104 will be described in more detail.

[0067] Image coding technologies such as VVC use a prediction process to predict the pixels of a block to be coded using pixels from a previously coded block in order to reduce the data volume of the coded bitstream while maintaining the image quality of the reproduced image. Prediction processes include intra-prediction, which uses pixels from a previously coded block in the same frame, and inter-prediction, which uses pixels from a block in a different coded frame. In addition, VVC standardizes a technique called RPR to enable decoding even when the resolution of the previously coded frame to be referenced and the frame to be coded are different.

[0068] Here, as an explanation of RPR, inter prediction when the resolution of a reference encoded frame and the encoding target frame differs will be further explained. The reference encoded frame is scaled to match the resolution of the encoding target frame, and inter prediction is performed. An example of a method for scaling the reference encoded frame is shown below. For simplicity, the explanation will be given for luminance values. As for chrominance, a similar discussion can be applied taking into account the number of samples, and therefore an explanation will be omitted. The prediction unit 104 is responsible for the following steps A1, A2, and A3.

[0069] (Step A1) The vertical scaling ratio scalingRatio[0] and the horizontal scaling ratio scalingRatio[1] are calculated. Hereinafter, scalingRatio[0] and scalingRatio[1] are collectively referred to as scalingRatio[x]. scalingRatio[x] is determined by the ratio between the size of the scaling window of the coded frame to be referenced and the size of the scaling window of the current frame to be coded. In this embodiment, the prediction unit 104 obtains this scalingRatio[x] from the generation unit 103 as RPR control information.

[0070] (Step A2) This function determines the interpolation filter used for scaling. For example, the coefficients of the interpolation filter are selected depending on the value of scalingRatio[x]. If scalingRatio[x] exceeds 1.75x, the coefficients in the table in Figure 3 are used. If scalingRatio[x] is less than 1.75x but exceeds 1.25x, the coefficients in the table in Figure 4 are used. In all other cases, the coefficients in the table in Figure 5 are used. The coefficients of the interpolation filter are determined by the sample position p to be calculated. p is an integer between 0 and 15, and is the numerator value when the minimum sample unit is divided into 1 / 16ths. For example, if sample point A and sample point (A+1) are divided into 16, the filter coefficients for the third sample point (A+3 / 16, p=3) are fL[3][i]=[-4, -1, 16, 29, 23, 7, -4, -2], referring to the table in Figure 3.

[0071] Hereinafter, when the filter coefficients obtained in this way are used in the horizontal direction, they will be written as fLH[p][i] (=fL[p][i]), and when they are used in the vertical direction, they will be written as fLV[p][i] (=fL[p][i]).

[0072] (Step A3) The reference image generated from the encoded frame to be referenced is resampled according to scalingRatio[x] so that it has the same resolution as the frame to be encoded. For example, the position of each pixel when the frame to be encoded is scaled by scalingRatio[x] is determined with 1 / 16 pixel accuracy. In addition, the reference image is interpolated by 16 times using the interpolation filter determined in step A2. For example, if scalingRatio[x] is 1.25 or more, a new sampling point (x3+p x ,y3+p y ) is calculated. Here, the coordinates of the pixel in question in the reference image are (x i ,y i ), the coordinates of the adjacent pixel on the left are (x (i-1) ,y i ), and the coordinates of the pixel adjacent below are (x i ,y (i-1)) notation is used. x , p y are integers modulo 16, and are indices of coordinates obtained by dividing the coordinates of adjacent pixels in the horizontal and vertical directions into 16 parts, respectively, and L(x, y) represents the luminance value of the coordinates (x, y). a is a normalization constant.

[0073]

number

[0074] Here, fLH[p][i] and fLV[p][i] are generated from the table in Figure 4. n are the coordinate values ​​from y0 to y7. The samples generated in this way are called the upsampled image. This upsampled image is thinned out to generate a resampled reference image with the same resolution as the frame to be coded from the reference image.

[0075] 6 and 7 are simplified diagrams for easily explaining the thinning method. The thinning method is the same for both the vertical and horizontal directions, so for simplicity, only the horizontal direction will be explained. It is assumed that both the reference image 601 and the encoding target frame 602 are composed of pixel blocks 603. It is assumed that each pixel block has a defined pixel value. That is, the reference image 601 is a 5x5 image, and the encoding target frame 602 is a 4x4 image. The origin is the upper left vertex of the entire image, and the coordinates of each pixel value are the coordinates of the upper left vertex of the pixel block. The coordinate values ​​of each pixel in the encoding target frame are multiplied by scalingRatio[x] (5 / 4 in the example of FIG. 6). Then, the x-coordinates of the encoding target frame 602 can be calculated as (0, 5 / 4, 5 / 2, 15 / 4) for (0, 1, 2, 3), respectively, and stored in, for example, H[x] of the coordinate array 703. Here, the image 605 is an image obtained by resampling the reference image 601 to have the same number of pixels as the encoding target frame 602. The pixel values ​​of the resampled image 605 may be constructed using pixel values ​​corresponding to the coordinates of the enlarged image from the upsampled image.

[0076] An example of the construction method will now be described with reference to Fig. 7. Reference number 701 is the pixel block (top row) at y=0 in image 605. Reference number 704 is the pixel block (top row) at y=0 in reference image 601.

[0077] Here, the luminance value of the reference image 601 is Y[x], and the luminance value of the resampled image 605 is Y'[x]. Note that x is the x-coordinate position of the luminance value. Then, to obtain the luminance value of coordinate position x of the resampled image 605, H[x] in the coordinate array 703 is referenced, the coordinate position of the reference image 601 is found, and the luminance value of that coordinate position can be obtained. When limited to the explanation of the x-coordinate position, it can be written as follows:

[0078] Y'[x]=Y[H[x]] For example, in pixel block 702, the x-coordinate position of the desired luminance value is 1, so x = 1 is set, H[1] = 5 / 4 is obtained from the coordinate array, and the value of Y[5 / 4] is used. Reference numeral 705 indicates a sampling point interpolated using the above filter between coordinate positions 1 and 2 in pixel block 704 where y = 0. Since the reference image has been interpolated in advance in 1 / 16 increments as shown by reference numeral 705, it is sufficient to obtain the luminance value of the coordinate corresponding to 5 / 4 as shown by reference numeral 706.

[0079] In this way, the resampled reference image of the same resolution can be constructed by upsampling the reference image and thinning out pixel values ​​other than those corresponding to the enlarged pixel positions of the frame to be coded.

[0080] Inter prediction is a process of predicting pixels of a block to be coded by referencing pixels of an encoded frame, or, if the number of pixels of the encoded frame differs from the number of pixels of the frame to be coded, pixels of a resampled image constructed using the above method. For simplicity, the encoded frame and the resampled image are collectively referred to as the inter prediction target image. For example, if there is no motion between the reference encoded frame and the inter prediction target image, the pixels of the block to be coded are predicted using pixels at the same positions in the inter prediction target image. In such a case, a (0, 0) motion vector indicating no motion is included in the prediction information. On the other hand, if there is motion between frames for the block to be coded, the motion vector (MVx, MVy) is included in the prediction information.

[0081] Returning to FIG. 1 , the operations of the transform / quantization unit 105, the inverse quantization / inverse transform unit 106, the image reproduction unit 107, and the in-loop filter unit 109 are as described above. The entropy coding unit 110 entropy codes, on a block-by-block basis, the quantization coefficients generated by the transform / quantization unit 105 and the prediction information generated by the prediction unit 104 to generate coded data. When coding the motion vector included in the prediction information, a difference between the motion vector of the block to be coded and the predicted motion vector calculated by the prediction unit 104, an identifier indicating which of the candidate predicted motion vectors has become the predicted motion vector, and the like are coded. The entropy coding method is not limited to a specific method, and Golomb coding, arithmetic coding, Huffman coding, and the like can be used.

[0082] The integrated coding unit 111 encodes the image analysis information generated by the image analysis unit 102 and the scaling window control information generated by the generation unit 103 to generate a frame resolution information code and a scaling window information code, respectively. The encoding method may be, for example, Golomb coding. The integrated coding unit 111 then generates header code data including the frame resolution information code and the scaling window information code, and multiplexes the header code data with the code data generated by the entropy coding unit 110 to generate a bitstream. The integrated coding unit 111 then outputs the generated bitstream.

[0083] The output destination of the bitstream generated in this way is not limited to a specific output destination. For example, the control unit 901 may output (store) the generated bitstream to the storage unit 906 or the memory unit 907, or may transmit the generated bitstream to the terminal device 1102 via the communication unit 908 and the network 803.

[0084] Thus, according to this embodiment, even when a zoom-in operation is performed on an imaging device having an optical zoom function, it is possible to improve the image quality of the captured frame and reduce the amount of code required for encoding using that frame.

[0085] In this embodiment, a high-resolution frame is defined as "an image having a number of pixels greater than the number of pixels of the image sensor of the image capturing unit 902," but this is not limited to this, and any frame having a higher resolution than the resolution of the frame generated in step S1003 may be used.

[0086] [Second embodiment] In the first embodiment, an imaging device 801 that changes various setting values ​​in response to instructions from a terminal device 802 was described, but it may also be configured to recognize a tracking subject in a frame and change the setting values ​​in response to the results of the recognition.

[0087] The numerical values, processing timing, processing order, processing subject, data (information) configuration / acquisition method / sending destination / sending source / storage location, etc. used in the above embodiment are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0088] In addition, some or all of the above-described embodiments may be used in appropriate combination, and some or all of the above-described embodiments may be selectively used.

[0089] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0090] The invention of this specification includes the following imaging device and imaging device control method. (Item 1) a generating means for capturing a moving image and generating frames; encoding means for performing encoding using the frames generated by the generating means; a determining means for determining whether or not to cause the generating means, which is generating a frame with a first resolution, to generate a frame with a second resolution higher than the first resolution when a change in the angle of view is instructed; Equipped with The encoding means When it is determined that the generating means is to generate a frame of the second resolution, the frame of the second resolution generated by the generating means is used as a long-term reference frame for encoding. An imaging device characterized by: (Item 2) The determination means 2. The imaging device according to item 1, characterized in that when an instruction to change the angle of view is given, if the difference between frames is less than a threshold, it is determined that the generation means should generate a frame with a second resolution. (Item 3) The determination means 2. The imaging device according to item 1, characterized in that when an instruction to change the angle of view is given, if the set value of the angle of view exceeds a specified limit value, it is determined that the generation means should generate a frame with a second resolution. (Item 4) The encoding means 4. The imaging device according to any one of items 1 to 3, wherein when it is determined that the generation means is to generate a frame of the second resolution, metadata is generated that describes settings for setting a scaling window for a frame before the start of change in angle of view, and metadata is generated that describes settings for using the frame of the second resolution as a long-term reference frame. (Item 5) The encoding means 5. The imaging device according to any one of items 1 to 4, characterized in that, when a change in the angle of view is not instructed, encoding is performed using a frame of the first resolution generated by the generation means. (Item 6) The encoding means 6. The imaging device according to any one of items 1 to 5, characterized in that, when it is not determined that the generating means should generate a frame of the second resolution, encoding is performed using the frame of the first resolution generated by the generating means. (Item 7) moreover, 7. The imaging device according to any one of items 1 to 6, further comprising a means for controlling the angle of view in response to the instruction. (Item 8) moreover, 8. The imaging device according to any one of items 1 to 7, further comprising a means for outputting the result of encoding by the encoding means to an external device. (Item 9) 9. The imaging device according to any one of items 1 to 8, wherein the encoding is VVC. (Item 10) A control method for an imaging device, comprising: a generation step in which a generation means of the imaging device captures a moving image and generates frames; an encoding step in which an encoding means of the imaging device performs encoding using the frames generated in the generating step; a determination step in which, when a change in the angle of view is instructed, a determination means of the imaging device determines whether or not to generate a frame with a second resolution higher than the first resolution in the generation step in which a frame with a first resolution is generated; Equipped with In the encoding step, When it is determined in the generating step that a frame of the second resolution is to be generated, the frame of the second resolution generated by the generating means is used as a long-term reference frame for encoding. 10. A method for controlling an imaging device, comprising:

[0091] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0092] 901: Control unit 902: Imaging unit 903: View angle control unit 904: Generation determination unit 905: Encoding unit 906: Storage unit 907: Memory unit 908: Communication unit

Claims

1. a generating means for capturing a moving image and generating frames; encoding means for performing encoding using the frames generated by the generating means; a determining means for determining, when a change in the angle of view is instructed, whether to cause the generating means, which is generating a frame with a first resolution, to generate a frame with a second resolution higher than the first resolution; Equipped with The encoding means When it is determined that the generating means is to generate a frame of the second resolution, the frame of the second resolution generated by the generating means is used as a long-term reference frame for encoding. An imaging device characterized by:

2. The determination means 2. The imaging device according to claim 1, wherein when an instruction to change the angle of view is given, if the difference between frames is less than a threshold value, it is determined that the generation means should generate a frame with the second resolution.

3. The determination means 2. The imaging device according to claim 1, wherein when an instruction to change the angle of view is given and the set value of the angle of view exceeds a specified limit value, it is determined that the generation means should generate a frame with the second resolution.

4. The encoding means 2. The imaging device according to claim 1, wherein, when it is determined that the generation means is to generate a frame of the second resolution, metadata is generated that describes settings for setting a scaling window for a frame before a change in the angle of view begins, and metadata is generated that describes settings for using the frame of the second resolution as a long-term reference frame.

5. The encoding means 2. The imaging device according to claim 1, wherein, when a change in the angle of view is not instructed, encoding is performed using frames of the first resolution generated by said generating means.

6. The encoding means 2. The imaging device according to claim 1, wherein, when it is not determined that the generating means should generate a frame with a second resolution, encoding is performed using the frame with a first resolution generated by the generating means.

7. moreover, The imaging device according to claim 1 , further comprising a means for controlling the angle of view in response to the instruction.

8. moreover, 2. The imaging apparatus according to claim 1, further comprising: means for outputting the result of encoding by said encoding means to an external device.

9. 2. The imaging device according to claim 1, wherein the encoding is VVC.

10. A control method for an imaging device, comprising: a generation step in which a generation means of the imaging device captures a moving image and generates frames; an encoding step in which an encoding means of the imaging device performs encoding using the frames generated in the generating step; a determination step in which, when a change in the angle of view is instructed, a determination means of the imaging device determines whether or not to generate a frame with a second resolution higher than the first resolution in the generation step in which a frame with a first resolution is generated; Equipped with In the encoding step, When it is determined in the generating step that a frame of the second resolution is to be generated, the frame of the second resolution generated by the generating means is used as a long-term reference frame for encoding.

10. A method for controlling an imaging device, comprising:

Images