Encoding device, encoding method, and computer program

Abstract

To provide an encoding device, an encoding method and a computer program capable of transmitting a plurality of image frames having different resolutions as one stream.SOLUTION: An encoding unit that uses a first type of image frame that uses intra-prediction and a second type of image frame that can use intra-prediction and inter-prediction, the encoding unit performing, when encoding a video including an image frame having a first resolution and an image frame having a second resolution different from the first resolution, inter-prediction after adjusting the resolution by enlarging or reducing the image frame to be referred to when referring to the image frame having the different resolution in the second type of image frame; When an image frame having a first resolution is encoded, if an immediately preceding image frame has a second resolution, inter prediction is performed by referring to another image frame having the first resolution without referring to the immediately preceding image frame.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to an encoding device, an encoding method, a computer program, and the like. [Background technology]

[0002] In video distribution, video is transmitted after compressing the data volume using a video encoding method such as H.264 / AVC (Advanced Video Coding) or HEVC (High Efficiency Video Coding).

[0003] Additionally, the Versatile Video Coding (VVC) coding method (hereafter referred to as VVC) has been standardized as a coding method for compressing and recording moving images. To improve coding efficiency, VVC incorporates a technology called Reference Picture Resampling (RPR). RPR is a technology that allows an image with a different resolution from the image being decoded to be used as a reference image, making it possible to change the resolution even in the case of inter-frame compression.

[0004] In Patent Document 1, a plurality of compressed videos with different specifications such as resolution and frame rate are generated and transmitted depending on the intended use of the video at the destination. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-270346 Summary of the Invention [Problem to be solved by the invention]

[0006] In the above-mentioned Patent Document 1, a problem arises in that a large amount of traffic is required by creating and transmitting a plurality of streams with different specifications depending on the purpose, and encoding / decoding processes take time. An object of the present invention is to provide an encoding device capable of transmitting a plurality of image frames with different resolutions as a single stream. [Means for solving the problem]

[0007] The encoding device according to claim 1 comprises: a first type of image frame utilizing intra prediction; a second type of image frame capable of using intra prediction and inter prediction; When encoding a video composed of an image frame having a first resolution and an image frame having a second resolution different from the first resolution, the encoding means When the second type image frame refers to an image frame having a different resolution, the referenced image frame is enlarged or reduced to match the resolution before performing inter prediction; When encoding an image frame of the first resolution, if the immediately preceding image frame is of the second resolution, inter-prediction is performed by referring to another image frame of the first resolution without referring to the immediately preceding image frame. [Effects of the Invention]

[0008] According to the present invention, a coding device is realized that can transmit a plurality of image frames with different resolutions as a single stream. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a functional block diagram illustrating an example of the configuration of a compressed data communication device 100 according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a functional block diagram illustrating an example of the configuration of an encoding unit 160 according to the first embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of a stream configuration according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing a filter coefficient table 1 according to the first embodiment. [Figure 5]FIG. 10 is a diagram showing a filter coefficient table 2 according to the first embodiment. [Figure 6] FIG. 3 is a diagram showing a filter coefficient table 3 according to the first embodiment. [Figure 7] 1(A) to 1(C) are simplified diagrams of a thinning method according to the first embodiment. [Figure 8] 10A and 10B are simplified diagrams of a method for thinning out one row according to the first embodiment. [Figure 9] 10 is a flowchart showing an example of a processing flow for generating a coded stream according to the first embodiment. [Figure 10] FIG. 2 is a diagram showing an example of a coded stream according to the first embodiment. [Figure 11] 1 is a functional block diagram showing an example of the configuration of a coded stream receiving device 1100 on a client side according to the first embodiment. [Figure 12] 10 is a flowchart showing an example of stream processing in the coded stream receiving device 1100 according to the first embodiment. [Figure 13] FIG. 10 is a diagram showing an example of generating a storage stream in the coded stream receiving device 1100 according to the first embodiment. [Figure 14] FIG. 10 is a diagram illustrating an example of generating an encoded stream and an example of communication with a client according to the second embodiment. [Figure 15] 11 is a flowchart showing an example of a recognition processing task in the coded stream receiving device 1100 according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions will be omitted or simplified.

[0011] (Embodiment 1) Fig. 1 is a functional block diagram showing an example of the configuration of a compressed data communication device 100 according to a first embodiment of the present invention. Note that some of the functional blocks shown in Fig. 1 are realized by causing a CPU or the like serving as a computer included in the compressed data communication device 100 to execute a computer program stored in a memory serving as a storage medium.

[0012] However, some or all of these functions may be implemented by hardware, which may be a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP).

[0013] 1 may not be contained in the same housing, but may be configured as separate devices connected to each other via signal paths. The above explanation regarding FIG. 1 also applies to FIG. 2.

[0014] Compressed data communication device 100 and a compressed data receiving device constitute a compressed data communication system. Reference numeral 101 denotes a CPU as a computer. Reference numeral 102 denotes a ROM (non-volatile memory) such as an EEPROM or flash memory. Reference numeral 103 denotes a RAM (volatile memory) such as an SRAM or DRAM.

[0015] Computer programs for realizing the functions according to this embodiment and data used when executing the computer programs are stored in ROM 102. These programs and data are appropriately loaded into RAM 103 via bus 110 under the control of CPU 101 and executed by CPU 101.

[0016] The imaging unit 120 includes a focus lens 121, a fixed lens 122, an aperture 123, and an image sensor 124. The focus lens 121 is moved along the optical axis by a lens driving unit 125. The aperture 123 is driven by the lens driving unit 125 to change the aperture size.

[0017] The image sensor 124 photoelectrically converts light that has passed through the focus lens 121 and the aperture 123 to generate an analog image signal. The generated analog image signal is subjected to sampling processing such as correlated double sampling, and then input to the camera signal processing unit 130.

[0018] The camera signal processing unit 130 converts the analog image signal into a digital image signal by A / D conversion, and then performs various digital image processing on the signal. The various digital image processing processes include offset processing, gamma correction processing, gain processing, RGB interpolation processing, noise reduction processing, contour correction processing, color correction processing, light source type determination processing, etc. The video signal after the digital image processing is stored in the RAM 103 via the bus 110.

[0019] The motor control unit 140 controls the lens driving unit 125. The video signal stored in the RAM 103 from the camera signal processing unit 130 undergoes resolution conversion in the scaling unit 150, and then undergoes compression processing by the encoding unit 160, generating a bitstream that is stored in the RAM 103 via the bus 110. In this embodiment, the image compression by the encoding unit 160 is performed based on the VVC standard, but is not limited to this.

[0020] The RPR control unit 170 determines the resolution of each frame created by the scaling unit 150, and controls the inter-frame reference state in the compression process by the encoding unit 160 as encoding means.

[0021] The IP communication unit 180 is connected to a network 181 via a LAN. The bit stream stored in the RAM 103 is distributed from the IP communication unit 180 through the network 181 to a compressed data receiving device.

[0022] 2 is a functional block diagram showing an example of the configuration of the encoding unit 160 according to embodiment 1. Reference numeral 200 denotes an image analysis unit that analyzes the angle of view value of an input frame and outputs the analysis results as image analysis information, and also outputs tile images that are combined with tile information for dividing the image into spatial regions based on image characteristics and external inputs.

[0023] Reference numeral 210 denotes an RPR control information generation unit, which generates, based on the RPR information received from the RPR control unit 170, information on the scaling ratio and offset position of the motion vector required for decoding using RPR (Reference Picture Resampling).

[0024] A prediction unit 220 performs intra-frame prediction, or inter-frame prediction, on tile-based image data to generate predicted image data. The prediction unit 220 also calculates and outputs a prediction error from the input image data and the predicted image data. It also outputs prediction information (such as a prediction mode and motion vectors) necessary for prediction, along with the prediction error.

[0025] A transform / quantization unit 230 performs orthogonal transform on the prediction errors output from the prediction unit 220 in units of blocks to calculate transform coefficients, performs quantization, and calculates quantized coefficients. A dequantization / inverse transform unit 231 dequantizes the quantized coefficients output from the transform / quantization unit 230 to regenerate transform coefficients, and further performs inverse orthogonal transform on them to regenerate prediction errors.

[0026] Reference numeral 250 denotes a frame memory that stores reconstructed image data. Reference numeral 240 denotes an image reconstruction unit that generates predicted image data by appropriately referencing the frame memory 250 based on the prediction information output from the prediction unit 220, and generates and outputs reconstructed image data from the input prediction error and predicted image data.

[0027] An in-loop filter unit 251 performs in-loop filtering such as deblocking filtering and sample adaptive offset on the reconstructed image, and outputs the filtered image.

[0028] An entropy coding unit 260 encodes the quantized coefficients output from the transform / quantization unit 230 and the prediction information output from the prediction unit 220 to generate and output coded data.

[0029] A bitstream generation unit 270 generates header code data by encoding the outputs from the image analysis unit 200 and the RPR control information generation unit 210. Furthermore, the bitstream generation unit 270 combines the header code data with the code data output from the entropy encoding unit 260 to generate and output a bitstream.

[0030] Next, the image encoding operation in the encoding unit 160 will be described below. In this embodiment, moving image data is input in frame units. The image analysis unit 200 receives image data for one frame and calculates a field of view change value. When an arbitrary frame is set as a reference frame, the field of view change value is the ratio of the field of view of the reference frame to the frame to be encoded.

[0031] Next, the RPR control information generator 210 signals by setting sps_ref_pic_resampling_enabled_flag of the SPS (Sequence Parameter Set) to 1 to indicate that RPR is to be used.

[0032] The number of pixels in the input frame is also calculated, and the number of pixels in the vertical direction and the horizontal direction for luminance are stored as pps_pic_width_in_luma_samples and pps_pic_height_in_luma_samples of the PPS (Picture Parameter Set), respectively.

[0033] The prediction unit 220 cuts the image data input from the image analysis unit 200 into a plurality of blocks and performs prediction processing on a block-by-block basis. As a result of the prediction processing, prediction errors are generated and input to the transformation and quantization unit 230. The prediction unit 220 also generates prediction information and outputs it to the image reproduction unit 240 and the entropy coding unit 260.

[0034] Here, we will explain in more detail the prediction process executed by the prediction unit 220 and the prediction information output from the prediction unit 220. In image coding techniques such as VVC, in order to reduce the amount of data in the coded bitstream while maintaining the image quality of the reproduced image, a prediction process is performed in which pixels of a coded block are used to predict pixels of the block to be coded.

[0035] Prediction processes include intra-prediction, which uses pixels from blocks that have already been coded within the same frame, and inter-prediction, which uses pixels from blocks in different coded frames. VVC also standardizes a technology called RPR, which allows decoding even when the resolution of the coded frame to be referenced differs from the resolution of the frame to be coded.

[0036] Here, as an explanation of RPR, inter prediction in the case where the resolution of a reference encoded frame and the resolution of a current encoding frame are different will be further explained.

[0037] FIG. 3 is a diagram showing an example of a stream configuration according to the first embodiment, illustrating an example where the resolution is changed midway through the stream using RPR. Frames 301 to 306 are arranged in chronological order. Frame 301 is an I-frame (I-picture) that is composed solely of intra-reference data. Here, an I-frame is a first type of image frame that utilizes intra-prediction.

[0038] Furthermore, frames 302 to 306 are P frames (P pictures (Predictive-Pictures)) that include a reference to the previous frame. Here, frames 301 to 303 constitute the first video.

[0039] Note that P frames are a second type of image frame that can use intra prediction and inter prediction, and the encoding unit 160 of this embodiment configures a GOP (Group Of Pictures) using the above-mentioned first type image frames and second type image frames.

[0040] At the timing of frames 303 to 304, the image is reduced to two-thirds vertically and horizontally. In terms of specific pixel count, for example, frames 301 to 303 are 1920 x 1080 pixels, while frames 304 to 306 are 1280 x 720 pixels. Here, frames 304 to 306 make up the second video.

[0041] In locations where a change in resolution occurs, the reference encoded frame is scaled (enlarged or reduced) to the same resolution as the frame to be encoded, and inter prediction is performed. An example of a method for scaling the reference encoded frame will be described below. For simplicity, only luminance values ​​will be described. A description of chrominance will be omitted, as it can be performed in a similar manner taking into account the number of samples.

[0042] (Step 1) The scaling ratios in the vertical and horizontal directions are calculated as scalingRatio[0] and scalingRatio[1], respectively. Hereinafter, scalingRatio[0] and scalingRatio[1] will be collectively referred to as scalingRatio[x].

[0043] The scalingRatio[x] is determined by the ratio between the size of the scaling window of the reference encoded frame and the scaling window of the encoding target frame. In this embodiment, the RPR control information generator 210 acquires the above ratio as RPR control information.

[0044] (Step 2) Next, an interpolation filter to be used for scaling is determined. That is, for example, coefficients of the interpolation filter are selected depending on the value of scalingRatio[x]. Fig. 4 shows filter coefficient table 1 of the first embodiment, Fig. 5 shows filter coefficient table 2 of the first embodiment, and Fig. 6 shows filter coefficient table 3 of the first embodiment.

[0045] If scalingRatio[x] is 1.75 times or more, the coefficients of the interpolation filter are used, for example, as shown in Table 1 in Figure 4. If scalingRatio[x] is less than 1.75 times but is 1.25 times or more, the coefficients are used, for example, as shown in Table 2 in Figure 5. If scalingRatio[x] is less than 1.25 times, the coefficients are used, for example, as shown in Table 3 in Figure 6.

[0046] The coefficients of the interpolation filter in each table are determined by the sample position p to be calculated. Tables 1 to 3 in Figures 4 to 6 are examples of filters defined for luminance expansion / contraction in the VVC standard. Table 1 is created based on the cutoff frequency when the ratio is 2x, and Table 2 is created based on the cutoff frequency when the ratio is 1.5x.

[0047] p is an integer ranging from 0 to 15, and is the numerator value when the smallest sample unit is divided into 1 / 16th units. For example, if sample point A and sample point A+1 are divided into 16, the filter coefficients for the third sample point (A+3 / 16, p=3) are fL[3][i]=[-4, -1, 16, 29, 23, 7, -4, -2], referring to Table 1.

[0048] Hereinafter, when the filter coefficients obtained in this way are used in the horizontal direction, they will be written as fLH[p][i] (=fL[p][i]), and when they are used in the vertical direction, they will be written as fLV[p][i] (=fL[p][i]).

[0049] (Step 3) Next, the reference image generated from the target encoding frame is resampled according to scalingRatio[x] so that it has the same resolution as the target encoding frame. For example, the position of each pixel when the target encoding frame is scaled by scalingRatio[x] is calculated with 1 / 16 pixel accuracy.

[0050] Also, the reference image is interpolated 16 times using the filter obtained in step 2. For example, if scalingRatio[x]≧1.25, a new sampling point (x3+px, y3+py) is obtained using the following equations 1 and 2.

[0051] Here, we use the notation that the coordinates of the target pixel in the reference image are (xi, yi), the coordinates of the adjacent pixel to the left are (x(i-1), yi), and the coordinates of the adjacent pixel below are (xi, y(i-1)).

[0052] Also, px and py are integers modulo 16 (divisor), and are indices of coordinates obtained by dividing the coordinates of adjacent pixels in the horizontal and vertical directions by 16, respectively, and L(x, y) represents the luminance value of coordinates (x, y). a is a normalization constant.

[0053]

number

[0054]

number

[0055] Here, fLH[p][i] and fLV[p][i] are generated from Table 2. Furthermore, yn is the coordinate value from y0 to y7. The samples generated in this manner are called upsampled images. This upsampled image is thinned out, and resampled from the reference image to the same resolution as the frame to be coded, and then a reference image is generated.

[0056] 7(A) to 7(C) are simplified diagrams of a thinning method according to embodiment 1, and Figures 8(A) and 8(B) are simplified diagrams of a thinning method for one line according to embodiment 1. The thinning method is the same for both the vertical and horizontal directions, so for simplicity, only the horizontal direction will be described.

[0057] In Fig. 7(A), 701 is a reference image, and in Fig. 7(B), 702 is a frame to be coded, which is composed of pixel blocks such as 703. Note that pixel values ​​are defined for each pixel block. In this case, the reference image 701 is a 6x6 image, and the frame to be coded 702 is a 4x4 image.

[0058] The origin is the upper left vertex of the entire image, and the coordinates of each pixel value are the coordinates of the upper left vertex of the pixel block. Here, the coordinate values ​​of each pixel in the encoding frame are multiplied by scalingRatio[x] (2 / 3 in the example of Figure 7).

[0059] Then, for the x coordinate of the encoding target frame 702 being (0, 1, 2, 3), the coordinate values ​​(0, 3 / 2, 6 / 2, 9 / 2) can be calculated and stored in, for example, a coordinate array H[x].

[0060] 7C, 704 denotes an image obtained by resampling the reference image to the number of pixels of the frame to be encoded. The pixel values ​​of each pixel block of the resampled image 704 may be constructed using pixel values ​​from the upsampled image that correspond to the coordinates of the enlarged image.

[0061] An example of the construction method will now be described with reference to Figure 8. 801 in Figure 8(A) shows only the pixel block at y=0 in the resampled image 704. Similarly, 804 in Figure 8(B) shows only the pixel block at y=0 in the reference image 701.

[0062] Here, the luminance value of the reference image 701 is Y[x], and the luminance value of the resampled image 704 is Y'[x]. Note that x is the x-coordinate value of the luminance value. Then, the luminance value of the coordinate position x of the resampled image 704 can be obtained by finding the coordinate position of the reference image 701 with reference to H[x] in the coordinate array 803. When only the x-coordinate is considered, it can be written as in Equation 3 below.

[0063]

number

[0064] For example, the coordinate of the desired brightness value for pixel block 802 is 1, so set x=1, find coordinate array H[1]=3 / 2, and obtain the value of Y[3 / 2]. 805 is a diagram that clearly shows the sampling points obtained by interpolating the coordinate values ​​1 and 2 of 804 using the filter described above.

[0065] Since the reference image has been interpolated in advance in 1 / 16 increments as shown in 805, it is sufficient to obtain the luminance value of the coordinate corresponding to 3 / 2 (=1 + 8 / 16) of 806. In this way, the reference image after resampling at the same resolution can be constructed by thinning out the pixel values ​​of the upsampled reference image except for those corresponding to the enlarged pixel positions of the frame to be coded.

[0066] Inter-prediction is a process in which, when the pixels of an encoded frame or the number of pixels of an encoded frame differ from the number of pixels of the frame to be encoded, pixels of the block to be encoded are predicted by referring to the pixels of the resampled image constructed using the above method.

[0067] For simplicity, the coded frame and the resampled image are collectively referred to as a target image for inter prediction. For example, if there is no motion between the coded frame to be referenced and the target image for inter prediction, pixels of the target block for coding are predicted using pixels at the same positions in the target image for inter prediction.

[0068] In such a case, the motion vector (0, 0), which indicates no motion, is included in the prediction information. On the other hand, if there is motion between frames for the block to be coded, the motion vector (MVx, MVy) is included in the prediction information.

[0069] Fig. 9 is a flowchart showing an example of a process flow for generating an encoded stream according to the first embodiment, and Fig. 10 is a diagram showing an example of an encoded stream according to the first embodiment. Note that the operation of each step in the flowchart of Fig. 9 is performed sequentially by a CPU or the like serving as a computer in the compressed data communication device 100 executing a computer program stored in a memory.

[0070] The process of generating a coded stream in the coding unit 160 under the control of the RPR control unit 170 will be described with reference to the flowchart in FIG. 9 and the example of the coded stream in FIG.

[0071] In step S900 of FIG. 9, depending on whether the next frame is an I frame or a P frame, the process proceeds to step S910 if it is an I frame, and to step S920 if it is a P frame.

[0072] In step S910, a reduced image with a relatively low resolution is created. That is, the video output from the camera signal processing unit 130 and stored in the RAM 103 is subjected to resolution conversion in the enlargement / reduction unit 150 to create a reduced image. At this time, a relatively low resolution is selected for use in storage on the client side (encoded stream receiving device 1100 side).

[0073] Then, in step S915, the encoding unit 160 performs intra-frame compression processing, and the result is temporarily stored in RAM 103. Then, the process proceeds to step S950. Note that the processing in steps S910 and S915 corresponds to the processing for generating I frames I0 and I1 in FIG.

[0074] In step S920, it is determined whether the image is intended for recognition. That is, if the image is intended for recognition on the client side, the determination is Yes and the process proceeds to step S930, whereas if the image is intended for storage on the client side, the determination is No and the process proceeds to step S940.

[0075] In the example shown in Fig. 10, high-resolution (second resolution) images (P1, P4, P7, P9) for recognition are generated once every three frames. For the other P frames, low-resolution (first resolution) images for storage are generated. However, this is just one example, and the resolution may be determined appropriately depending on other applications.

[0076] As shown in FIG. 10, the encoding step of this embodiment uses a first type of image frame that uses intra prediction and a second type of image frame that can use both intra prediction and inter prediction.

[0077] In step S930, a reduced image with a relatively high resolution is created. That is, the image output from the camera signal processing unit 130 and stored in the RAM 103 is subjected to resolution conversion in the enlargement / reduction unit 150. The resolution at this time is used for recognition on the client side, so a relatively high resolution is selected.

[0078] 10, P frames P1, P4, P7, and P9 are generated in step S903. In the next step S933, the immediately preceding image frame is selected as a reference image. In step S935, encoding unit 160 performs frame reference compression processing with reference to the immediately preceding frame, and the result is temporarily stored in RAM 103, after which the process proceeds to step S950.

[0079] In the frame reference compression process in step S935, RPR is used because the resolution of the immediately preceding frame is lower than that of the frame being compressed. That is, the decoded result of the immediately preceding frame is enlarged to match the resolution of the frame being compressed, and then frame reference is performed.

[0080] Since a low-resolution image is enlarged and used for reference, the difference becomes large and the compressed size tends to increase, but since the client's use is for recognition, this control is performed to increase the recognition rate.

[0081] In this embodiment, when encoding a video composed of an image frame of a first resolution and an image frame of a second resolution different from the first resolution, the image frame of the second type is referenced. In this case, the referenced image frame is enlarged or reduced to match the resolution, and then inter-prediction is performed and the video is encoded using RPR.

[0082] In step S940, a reduced image with a relatively low resolution is created. That is, the image output from the camera signal processing unit 130 and stored in the RAM 103 is subjected to resolution conversion by the enlargement / reduction unit 150. Since the resolution at this time is used for storage on the client side, a relatively low resolution is selected.

[0083] In the example shown in Fig. 10, P0, P2, P3, P5, P6, P8, and P10 are created in step S940. In the next step S943, the nearest frame with the same resolution is selected as the reference image. In the example shown in Fig. 10, the previous frame is selected as the reference image for P0, P3, P6, and P8, and the frame two frames before that is selected as the reference image for P2, P5, and P10.

[0084] In this embodiment, if the immediately preceding frame is a high-resolution frame for recognition, it is restricted from being referenced. This is to improve convenience when a high-resolution frame for recognition is replaced with a low-resolution frame for storage in the client-side processing described later.

[0085] In this embodiment, when encoding an image frame of a first resolution, if the immediately preceding image frame is of a second resolution, the encoding is performed by performing inter-prediction by referring to other image frames of the first resolution, without referring to the immediately preceding image frame.

[0086] In step S945, the encoding unit 160 performs frame reference compression processing using the reference image selected in step S943, and the result is temporarily stored in the RAM 103, after which the process proceeds to step S950.

[0087] In step S950, the compressed data is transmitted. That is, the data compressed in steps S915, S945, and S935 is read from RAM 103 and transmitted from IP communication unit 180.

[0088] In step S960, it is determined whether the series of compression processes has ended, and if it is still ongoing, the process returns to step S900. If it is determined in step S960 that the series of compression processes has ended, the process flow in FIG.

[0089] Fig. 11 is a functional block diagram showing an example of the configuration of a client-side coded stream receiving device 1100 according to embodiment 1. Note that some of the functional blocks shown in Fig. 11 are realized by causing a CPU 1130 or the like serving as a computer included in the coded stream receiving device 1100 to execute a computer program stored in a memory serving as a storage medium.

[0090] However, some or all of these functions may be implemented by hardware. Examples of hardware that can be used include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs). Furthermore, the functional blocks shown in Fig. 11 do not have to be built into the same housing, and may be configured as separate devices connected to each other via signal paths.

[0091] The coded stream receiving device 1100 functions as a compressed data receiving device. The coded stream receiving device 1100 as a compressed data receiving device constitutes a compressed data communication system together with the compressed data communication device 100.

[0092] In FIG. 11, 1130 is a CPU as a computer, 1140 is a ROM (non-volatile memory) such as an EEPROM or flash memory, and 1150 is a RAM (volatile memory) such as an SRAM or DRAM.

[0093] Computer programs for realizing the functions of the coded stream receiving device 1100 according to this embodiment and data used when the programs are executed are stored in a ROM 1140. These programs and data are appropriately loaded into a RAM 1150 via a bus 1110 under the control of a CPU 1130, and are executed by the CPU 1130.

[0094] The IP communication unit 1120 is connected to a network 1121 via a LAN or the like. The bit stream delivered from the compressed data communication device 100 via the network 1121 is stored in the RAM 1150 by the IP communication unit 1120.

[0095] The bitstream stored in the RAM 1150 is decoded by the encoding / decoding processing unit 1170, and the decoded video is stored in the storage unit 1190. The storage unit 1190 is a large-capacity storage device such as a hard disk or SSD (Solid State Drive).

[0096] Of the images stored in the storage unit 1190, frames with a high resolution for recognition are sent to the recognition processing unit 1180, where image recognition processing is performed.

[0097] The frames used for image recognition processing are reduced to a storage resolution by the scaling unit 1160, and after encoding processing by the encoding / decoding processing unit 1170, they are replaced with frame data of a larger resolution for image recognition that was stored in the storage unit 1190.

[0098] FIG. 12 is a flowchart showing an example of stream processing in the coded stream receiving device 1100 according to the first embodiment, and FIG. 13 is a diagram showing an example of generating a storage stream in the coded stream receiving device 1100 according to the first embodiment. The manner in which a received stream is processed according to the flowchart in Fig. 12 will be explained using the example stream in Fig. 13. The distribution data in the upper part of Fig. 13 is the same as that in Fig. 10 used in the explanation of the distribution side.

[0099] The operations of the steps in the flowchart of FIG. 12 are performed sequentially by a CPU or the like serving as a computer within the coded stream receiving device 1100 executing a computer program stored in memory.

[0100] In step S1200, one frame is decompressed and saved. That is, one frame of data is extracted from the bitstream stored in RAM 1150, and the data is decoded by encoding / decoding processing unit 1170, and the decoded video is stored in storage unit 1190.

[0101] In step S1210, it is determined whether the frame data processed in step S1200 is high-resolution data for recognition, and if it is, the subsequent steps S1220 to S1250 are performed. In Figure 13, P1, P4, P7, and P9 are determined to be high-resolution data for recognition.

[0102] In step S1220, the frame data for recognition is copied to RAM 1150, and recognition processing is started by recognition processing unit 1180. Note that the processing from step S1230 onwards is performed in parallel with the recognition processing started in step S1220.

[0103] In step S1230, the large-resolution frame data for recognition temporarily stored in RAM 1150 is reduced by enlargement / reduction unit 1160 to a resolution for storage.

[0104] In step S1240, the image reduced in step S1230 is encoded by the encoding / decoding processing unit 1170 using the immediately preceding frame data as a reference frame.

[0105] In step S1250, the data encoded in step S1240 is replaced with high-resolution frame data for recognition stored in storage unit 1190.

[0106] That is, as shown in Fig. 13, the high-resolution frame data for recognition, P1, P4, P7, and P9, are replaced with reduced data P1', P4', P7', and P9', respectively. Note that since the frames P1, P4, P7, and P9 are generated as frames that are not referenced by subsequent frames, replacing only these frames does not affect the subsequent frames.

[0107] In step S1260, it is determined whether the series of stream processing has ended, and if it is still ongoing, the process returns to step S1200. If it is determined in step S1260 that the series of stream processing has ended, the process flow in FIG. 12 ends.

[0108] As described above, according to this embodiment, a stream with a high frame rate and a low frame rate and a different resolution are synthesized using RPR and coded as a single stream for distribution.

[0109] Furthermore, since multiple image frames with different resolutions can be transmitted as a single stream depending on the application, the traffic load is reduced compared to transmitting multiple streams, and the time required for encoding / decoding is also reduced.

[0110] (Embodiment 2) 14 is a diagram showing an example of generation of a coded stream and an example of communication with a client according to embodiment 2. The functional blocks and basic operations of embodiment 2 are the same as those of embodiment 1, and therefore a description thereof will be omitted.

[0111] In the first embodiment, a high-resolution image for recognition is generated once every three frames, but in this embodiment, a high-resolution image for recognition is generated and transmitted upon receiving a request from a client (encoded stream receiving device 1100). The request from the client also includes information regarding the resolution required for recognition.

[0112] That is, in this embodiment, the encoding means generates an image frame with the second resolution based on a request from the receiving side (encoded stream receiving device 1100). Also, the encoding means generates an image frame with the second resolution at a timing based on the request from the receiving side (encoded stream receiving device 1100) and transmits it to the receiving side.

[0113] Moreover, the image frames with the second resolution are used for image recognition on the receiving side (encoded stream receiving device 1100), and the image frames with the first resolution are used for storage on the receiving side.

[0114] In Figure 14, the frame received as P1 is recognized by R1 in the encoded stream receiving device 1100, and then the resolution required for the next recognition is calculated and a request is sent from the encoded stream receiving device 1100 to the compressed data communication device 100.

[0115] In response to this request, the compressed data communication device 100 creates frame P4 and transmits it to the coded stream receiving device 1100. The coded stream receiving device 1100 receives P4, recognizes it using R4, and transmits a request for an image with a specified resolution to the compressed data communication device 100. The compressed data communication device 100 then creates P6 and transmits it to the coded stream receiving device 1100.

[0116] As a result of the recognition processing of R6 in the coded stream receiving device 1100, a request to create an image of a predetermined resolution is again sent to the compressed data communication device 100, and P9 is created.

[0117] Fig. 15 is a flowchart showing an example of a recognition processing task in the coded stream receiving device 1100 according to embodiment 2. Note that the operation of each step in the flowchart in Fig. 15 is performed sequentially by a CPU or the like serving as a computer in the coded stream receiving device 1100 executing a computer program stored in a memory.

[0118] In step S1500, it is determined whether an instruction to start the recognition process has been issued in step S1220 of Fig. 12. If the determination is No, the process returns to step S1500. If the determination is Yes, the process proceeds to step S1510, where the recognition process is executed. The time required for this recognition process may vary greatly depending on the content of the recognition process. For example, in person detection, it may take a long time if there are many people on the screen.

[0119] In step S1520, the resolution required for the next recognition is calculated. This is because, for example, when performing recognition to track an object within the screen, if the object to be tracked moves away from the camera and becomes smaller in image, a higher resolution is required for recognition. Conversely, if the object approaches, for example, recognition is possible with a lower resolution.

[0120] In step S1530, a request to transmit an image frame for recognition at the resolution calculated in step S1520 is sent to the server side (compressed data communication device 100).

[0121] In step S1540, it is determined whether the recognition task has been completed, and if the determination is No, the process returns to step S1500, whereas if the determination is Yes, the process flow in FIG. 15 is terminated.

[0122] As described above, this embodiment has the advantage of being able to flexibly respond to cases where the time required for recognition processing performed on the client side (encoded stream receiving device 1100) or the resolution required for recognition changes.

[0123] The present invention has been described above in detail based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible based on the spirit of the present invention, and these are not excluded from the scope of the present invention.

[0124] The present invention also includes those that realize the functions of the above embodiments using, for example, at least one processor such as a CPU, memory, or circuit (for example, ASIC). Also, multiple processors may be used to perform distributed processing.

[0125] In order to realize part or all of the control in the above-described embodiments, a computer program that realizes the functions of the above-described embodiments may be supplied to an encoding device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the encoding device or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. The present invention also includes the following combinations.

[0126] (Configuration 1) An encoding device having an encoding means that uses a first type of image frame that utilizes intra prediction and a second type of image frame that can utilize intra prediction and inter prediction, wherein when encoding video composed of an image frame of a first resolution and an image frame of a second resolution that is different from the first resolution, if the second type of image frame references an image frame of a different resolution, the encoding means enlarges or reduces the referenced image frame to match the resolution before performing inter prediction, and when encoding an image frame of the first resolution, if the immediately preceding image frame is of the second resolution, the encoding means performs inter prediction by referring to another image frame of the first resolution without referring to the immediately preceding image frame.

[0127] (Configuration 2) The encoding device according to configuration 1, wherein the second resolution is greater than the first resolution, and the encoding means generates the image frame of the second resolution based on a request from a receiving side.

[0128] (Configuration 3) The encoding device according to Configuration 2, wherein the second resolution is greater than the first resolution, and the encoding means generates the image frame of the second resolution at a timing based on a request from the receiving side and transmits it to the receiving side.

[0129] (Configuration 4) The encoding device according to configuration 2 or 3, wherein the image frame of the second resolution is used for image recognition on the receiving side.

[0130] (Configuration 5) The encoding device according to any one of configurations 2 to 4, wherein the image frame of the first resolution is used for storage on the receiving side.

[0131] (Configuration 6) The encoding device described in any one of configurations 1 to 5, characterized in that when the second type image frame references an image frame of a different resolution, the encoding means enlarges or reduces the referenced image frame to match the resolution, and then encodes it using RPR when performing inter-prediction.

[0132] (Method) An encoding method that uses a first type of image frame that utilizes intra-prediction and a second type of image frame that can utilize intra-prediction and inter-prediction, wherein when encoding video composed of an image frame of a first resolution and an image frame of a second resolution that is different from the first resolution, if the second type of image frame references an image frame of a different resolution, the referenced image frame is enlarged or reduced to match the resolution before performing inter-prediction, and when encoding an image frame of the first resolution, if the immediately preceding image frame is of the second resolution, inter-prediction is performed by referring to another image frame of the first resolution without referencing the immediately preceding image frame.

[0133] (Program) A computer program for controlling the encoding means of the encoding device according to any one of configurations 1 to 6 by a computer. [Explanation of symbols]

[0134] 100: Compressed data communication device 150: Zoom section 160: Encoding section 170: RPR control unit

Claims

1. a first type of image frame utilizing intra prediction; a second type of image frame capable of using intra prediction and inter prediction; When encoding a video composed of an image frame having a first resolution and an image frame having a second resolution different from the first resolution, the encoding means When the second type image frame refers to an image frame having a different resolution, the referenced image frame is enlarged or reduced to match the resolution before performing inter prediction; An encoding device characterized in that, when encoding an image frame of the first resolution, if the immediately preceding image frame is of the second resolution, inter-prediction is performed by referring to another image frame of the first resolution without referring to the immediately preceding image frame.

2. 2. The encoding device according to claim 1, wherein the second resolution is greater than the first resolution, and the encoding means generates the image frames at the second resolution based on a request from a receiving side.

3. The encoding device according to claim 2, characterized in that the second resolution is greater than the first resolution, and the encoding means generates the image frame of the second resolution at a timing based on a request from the receiving side and transmits it to the receiving side.

4. 3. The encoding device according to claim 2, wherein the image frames at the second resolution are used for image recognition at the receiving end.

5. 3. The encoding device of claim 2, wherein the image frames of the first resolution are used for storage at the receiving end.

6. The encoding device described in claim 1, characterized in that when the second type image frame references an image frame of a different resolution, the encoding means enlarges or reduces the referenced image frame to match its resolution and then encodes it using RPR when performing inter prediction.

7. a first type of image frame utilizing intra prediction; a second type of image frame for which intra prediction and inter prediction can be used, the method comprising: When encoding a video composed of an image frame having a first resolution and an image frame having a second resolution different from the first resolution, When the second type image frame refers to an image frame having a different resolution, the referenced image frame is enlarged or reduced to match the resolution before performing inter prediction; an encoding method characterized in that, when encoding an image frame of the first resolution, if the immediately preceding image frame is of the second resolution, inter prediction is performed by referring to another image frame of the first resolution without referring to the immediately preceding image frame.

8. A computer program for controlling the encoding means of the encoding device according to any one of claims 1 to 6 by a computer.

Images